COLA: Towards Efficient Multi-Objective Reinforcement Learning with Conflict Objective Regularization in Latent Space

We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper.

We will try our best to address each of the concerns and questions raised by the reviewer below:

1. [Re: How did you conduct the hyperparameter search? What was the budget? What was the search space? Optimizing which metric?]

I primarily focus on two core metrics: HV and UT. The evaluation budget is typically set to 3 million steps. The search space consists of several hyperparameters:

• c ∈ {0.25, 0.0}
• β ∈ {0.001, 0.01, 0.1, 1.0}

Notably, c = 0.25 is kept consistent across all experiments.

2. [Re: To what kind of latent size are you compressing the states (z)? And what is the size of c?]

We set latent size to 50 for both z and c.

3. [Re: Fig 2: I think s_t is missing as input in the bottom model.]

Thanks to the reviewer for pointing out this oversight. We will address it in the revised version.

4. [Re: Please consider adding the values of the hyperparameters in the appendix for reproducibility.]

We provide the detailed hyperparameter settings in the appendix; see Appendix C and Table 2 for more information. We will also release the code to ensure reproducibility.

5. [Re: I would appreciate seeing a comparison vs. GPI and MORL/D, and on different environments than the Mujocos, so we have a better view and understanding of the algorithm’s limits.]

We thank the reviewer for the valuable suggestion. We conduct evaluations on two continuous action space tasks from MO-Gym. Both sets of experiments are carried out using 100k environment steps.

These results indicate that COLA exhibits strong or competitive performance on such tasks.

6. [Re: Did you run experiments where you condition only with c (not z ) to see if z is adding anything at all? I suspect it should not, since it is essentially an encoded s?]

To address the reviewer's concern, we conduct an ablation study using only c while removing z.

The experimental results are shown in the table below:

We observe that incorporating z leads to better performance compared to the variant without z. This improvement is primarily due to the fact that the encoder extracts higher-level latent information from the original state s, which helps accelerate the approximation of the critic function and thereby improves learning efficiency.

7. [Re: For the COR loss (line 184): how is this different from Envelope in the end? It might be nice to have the two losses side by side in the appendix to clearly be able to see where they differ.]

The COR loss is not necessarily tied to the Envelope framework and can be applied to any MORL algorithm. It can be directly added to the original value function loss as an auxiliary term.

8. [Re: I think the paper could benefit from a bit more experiments: on different environments than Mujoco.]

In addition to the experiments mentioned above, we also conduct further evaluations on two other domains:

• Further experiments on the DeepMind Control Suite, using image-based inputs (800.000 environment steps and 5 seeds)

The objectives for these tasks include energy consumption and the original task objectives, such as running speed, walking speed, and jumping height. In this case, we combine COLA with RAD and compared it against Envelope-RAD with 800.000 environment steps. The experimental results are shown as follows:

• Experiment on the DeepSea Treasure task from Q-Pensieve with 150.000 environment steps and 5 seeds

The experiments above demonstrate that COLA can further improve the performance across both HV and UT, whether in image input tasks or other types of tasks.

---

We hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. We are always willing to answer any of the reviewer's concerns about our work and we are looking forward to more inspiring discussions.

We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper.

We will try our best to address each of the concerns and questions raised by the reviewer below:

1. [Re: There is no comment regarding this in the paper nor an ablation experiment exploring how different regimes for line 4 of the algorithm (when the random set of weights is chosen) would affect performance (Fig 5 shows only uniform sampling and perhaps not even shown in the best way). What are the assumptions behind the schedule for the set of weights using in the training so that the regularization proposed works as expected?]

We thank the reviewer for the helpful suggestion. In our current implementation, we sample a single preference w_i at each step and perform training based on it. This strategy follows the prior work Q-Pensieve, and we did not modify it in order to ensure a fair comparison.

That said, we agree that more efficient preference sampling strategies may exist. To explore this, we conducted additional experiments using an alternative approach: instead of sampling only one preference vector during training, we sample multiple preferences instead of one preference.

Overall, we find that sampling a single preference yields better performance in most tasks. This aligns with recent findings suggesting that multiple training iterations on the same batch lead to improved outcomes. Fully learning under one preference before switching to another appears more beneficial for training.

2. [Re: Also related, all the evaluations presented are done in Mujoco tasks, which effectively means there is just a single evaluation domain, it would be good to have some other different domains included to make sure the gain of performance is not only due artifacts of this domain.]

• Further experiments on the DeepMind Control Suite, using image-based inputs (800.000 environment steps and 5 seeds)

The objectives for these tasks include energy consumption and the original task objectives, such as running speed, walking speed, and jumping height. In this case, we combine COLA with RAD and compared it against Envelope-RAD with 800.000 environment steps. The experimental results are shown as follows:

• Experiment on the DeepSea Treasure task from Q-Pensieve with 150.000 environment steps and 5 seeds

• Experiment on the two tasks from MO-Gym with 10.000 environment steps and 5 seeds

The experiments above demonstrate that COLA can further improve the performance across both HV and UT, whether in image input tasks or other types of tasks.

3. [Re: where is PGMORL's results in Figure 3?]

We sincerely apologize for overlooking this issue. The comparison table is provided below, and we will update the curves in a revised version.

For all evaluation results, including Pareto front evaluations, we assess the performance after training is complete, using the trained policy. The full details of the evaluation procedure are provided in Appendix B.1.

5. [Re: For halfcheetah it seems like CAPQL has discovered some policies that way more efficient for speed forward values of around 2000. This could result in significantly superior policies depending on the scenario (if the application mandates power efficiency should be >=2000, for example, CAPQL's policy would be better. This goes against all other results presented, I would expect some explanation of in which conditions this behavior could arise, because it could affect where the new proposed method is better or not. ]

In our experiments, we observed that in the HalfCheetah task, COLA initially achieves better performance in speed. However, COLA must maintain a significantly larger Pareto front compared to other methods. During training, this requirement can lead to a phenomenon where performance under certain preferences improves as a result of maximizing cumulative returns, while performance under other preferences degrades over time. We believe this may be attributed to a form of catastrophic forgetting, preventing the algorithm from consistently maintaining the best-found performance across all objectives. This issue may require additional constraint mechanisms to mitigate.

Therefore, if the goal is to satisfy a specific requirement—e.g., power efficiency ≥ 2000—this constraint should be explicitly incorporated into the optimization objective, such as by modifying the reward function accordingly. Doing so typically leads to better and more targeted results.

---

We hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. We are always willing to answer any of the reviewer's concerns about our work and we are looking forward to more inspiring discussions.

We are very pleased to have addressed all of the reviewers’ concerns and are deeply grateful for your positive evaluation of our work and continued support.

Your thoughtful feedback has been instrumental in improving our work, and we will incorporate all relevant details from the discussions into the revised version.

In addition, we guarantee that the code will be made publicly available at the time of the paper’s release to ensure reproducibility.

Thank you once again for your valuable time and constructive feedback.

We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper.

We will try our best to address each of the concerns and questions raised by the reviewer below:

1. [Re: The rationale for incorporating the latent state-action representation into the inputs of the value function is unclear. From Eq. (3), it appears that the function estimates the expectation of latent representation of the subsequent state resulting from action. However, the benefit and necessity of including this representation in the Q-function inputs remain unclear and intuitively unconvincing for me. ]

Through OADM, the latent state-action representation c captures an important piece of information—dynamic information. In locomotion tasks such as those in MuJoCo, it is difficult to accurately estimate the next state based solely on the current state and action. However, this missing information can often be implicitly captured and supplemented by the latent representation c.

Based on the state-action representation c, the value network effectively gains access to an implicit prediction of the next state. This can be seen as a one-step model rollout, helping the network better understand how a given state-action pair influences long-term returns under the current preference, and thus produce more accurate value estimates.

To address the reviewer's concern, we conduct ablation studies on c across four tasks mentioned above (5 runs, 2M steps): Envelope (UT 1.0, HV 1.0), COLA (UT 1.29, HV 2.14), and COLA w/o c (UT 1.15, HV 1.37). We observe that removing c reduces performance due to absent temporal information, which hinders future state predictions and value approximation.

2. [Re: Regularizing by sampling a single random preference within each iteration may be inefficient, potentially leading to instability or convergence issues.]

To address the reviewer’s concern, we follow CAPQL and sample a batch preferences each step. The results are shown below.

Overall, we find that sampling a single preference yields better performance in most tasks. This aligns with recent findings suggesting that multiple training iterations on the same batch lead to improved outcomes. Fully learning under one preference before switching to another appears more beneficial for training.

3. [Re: Methodological novelty is somewhat limited. Both latent space modeling of states and state-action transitions, as well as regularization approaches for managing conflicting objectives, are common designs in ML literature.]

COLA targets two key challenges in MORL: the difficulty of knowledge sharing and the presence of conflicting objectives. To address these issues, we propose OADM and COR. OADM is designed to extract temporal dynamic information, while COR serves to constrain conflicting objectives and approximate the value function. Our goal is to provide a simple yet effective MORL algorithm, supported by extensive experimental evidence. In addition to the experiments presented in the main text, we further validate the effectiveness of our approach across several additional domains.

• experiments on the DeepMind Control Suite, using image-based inputs (800.000 environment steps and 5 seeds)

• Experiment on the DeepSea Treasure task from Q-Pensieve with 150.000 environment steps and 5 seeds

• Experiment on the two tasks from MO-Gym with 10.000 environment steps and 5 seeds

---

We hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. We are always willing to answer any of the reviewer's concerns about our work and we are looking forward to more inspiring discussions.

We are pleased to address the reviewers’ concerns through additional experimental investigations.

We would like to offer the following additional clarifications and responses.

1. Regarding a single randomly sampled preference during training, this design choice was guided by empirical observations indicating more efficient optimization. We hypothesize that this benefit arises from allowing the network to more fully exploit each batch with respect to a single preference, thereby enabling better adaptation to its associated distribution. Similar observations have been made in single-objective RL, where reusing batch data has been shown to improve performance and is supported by convergence guarantees [1].

2. Regarding novelty,

   • The idea of temporal representation originally emerged in the context of single-agent RL for better sample efficiency. In this work, we adapt and extend it to address challenges unique to MORL. Specifically, we remove the reward prediction component and detach the return gradient to prevent interference of different preferences, and further enhance the representation space by concatenating state and state-action embeddings. To the best of our knowledge, this is the first successful application of temporal representation learning in the MORL setting.
   • Regarding the regularization term, our approach is most closely related to prior work in multi-task learning that addresses conflicts between different loss functions by directly modifying gradients. In contrast, we focus on value function approximation in MORL and propose a method that mitigates the adverse effects of conflicting optimization objectives. We achieve this through a simple yet effective regularization term. To the best of our knowledge, the use of regularization for resolving conflicts in value function approximation has not been specifically investigated in MORL.

   Thanks to its above components, COLA is designed to be simple yet effective. We have validated its effectiveness across 16 tasks with 3 different domains.

We are committed to releasing the code to ensure the reproducibility of our results.

If there are any remaining questions regarding the experimental design or other aspects of our work, we would be grateful for the opportunity to further clarify. We sincerely appreciate the reviewers’ time and thoughtful feedback.

[1]: Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse.

We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper.

We will try our best to address each of the concerns and questions raised by the reviewer below:

1. [Re: What is the per-iteration training overhead introduced by COR compared to baselines?]

We thank the reviewer for the valuable suggestion. To address the concern, we have conducted the runtime experiments. The results show that COR incurs an additional time overhead of approximately 12.9% with one single CPU.

2. [Re: Is there a theoretical motivation for choosing c = 0.25? Could it be task-specific? Would an adaptive or learned threshold perform better? ]

In all experiments, c is set to 0.25. A smaller value of c indicates that the COR loss has less influence, whereas c = 1.0 means that COR is always active, regardless of whether conflicts are present or not. We initially conduct a preliminary comparison between c = 0.0 and c = 0.25 on a single case and find that c = 0.25 yields the best performance. Based on this observation, we adopt c = 0.25 consistently across all experiments.

We also conducted an ablation study on the effect of c, as shown below. The results indicate that c = 0.25 generally delivers strong performance across different tasks.

We have not yet identified a robust solution for the automatic adjustment of c. One potential approach is to introduce heuristic methods, such as evolutionary algorithms, to explore and iteratively refine multiple c values. However, we have observed that c = 0.25 consistently serves as an effective choice across a wide range of tasks. For example, the following two domains, COLA also uses c = 0.25, further demonstrating the generality of this parameter choice.

• experiments on the DeepMind Control Suite, using image-based inputs (800.000 environment steps and 5 seeds)

• Experiment on the DeepSea Treasure task from Q-Pensieve with 150.000 environment steps and 5 seeds

• Experiment on the two tasks from MO-Gym with 10.000 environment steps and 5 seeds

3. [Re: How does the computational complexity of COR scale with the number of objectives or preference dimensions?] The complexity of COR remains unchanged as the number of optimization objectives increases. This is because, regardless of how many objectives are involved, the optimization procedure still samples a pair of preference vectors (w_i, w_j) from the preference space for each batch of data and computes the COR loss based on Equation (7). Therefore, the computational complexity does not grow with the number of objectives.

4. [Re: The paper attributes instability from discarding raw states s_t to “real-time updates of OADM” (p.4). Could you provide a deeper analysis or diagnostics to clarify this issue?]

This is primarily based on empirical observations. We find that omitting s causes performance fluctuations in some tasks, such as Hopper-2d. The HV and UT results are as follows:

We observe that excluding s leads to performance drops in learning process. Moreover, using s as input is more efficient.

---

We hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. We are always willing to answer any of the reviewer's concerns about our work and we are looking forward to more inspiring discussions.