Thanks for taking the time to provide feedback on our paper. We appreciate your valuable comments and would like to address each of your concerns.

1. Could you provide an additional ablation study on the influence of the number of tasks? Specifically, does the representation for task A differ when learning tasks {A, B} together versus tasks {A, B, C} together? Additionally, does the number of tasks impact the accuracy of task A?

Thanks for your insightful suggestion. We have conducted an ablation study on the number of tasks $K$ and compared the performance on five tasks. The results including average return and average success rate across tasks are shown below.

From these results, it can be observed that as the number of tasks $K$ increases from 3 to 5 to 10, both the average return and success rate in experiments gradually increase. This suggests that when $K$ is not large, increasing the number of tasks $K$ can enhance the performance of a specific task. We attribute this to the speculation that as the number of tasks increases, the representation learning for a specific task improves. As mentioned in Section 3.1, our method learns a task-specific preference representation and an optimal representation by constructing positive and negative samples, where the negative samples include all trajectories from other tasks. When $K$ increases, the number of negative samples significantly increases, which helps our method learn the preference representation under the given task. Therefore, learning from more tasks leads to better representations of a specific task, resulting in improved performance and accuracy for that particular task.

It is also important to note that when $K$ becomes too large, the learning process for the diffusion model becomes more challenging. This is because the diffusion model needs to fit the trajectory distributions of multiple tasks. Especially when the multi-task trajectory data are low-quality, the difficulty of fitting for the diffusion model increases.

2. Could you highlight the best results in each row of Table 1? It appears that CAMP's performance is not consistently better than existing methods across these environments, and its average result is not particularly strong. Does this imply that CAMP struggles to learn a good task representation? Alternatively, do you have any other explanations for these observations?

Thanks for your feedback. We will highlight the best results in each row of Table 1 to clarify the comparisons in the next version. We notice that your concern is similar to the second question raised by Reviewer 3Xmu. Therefore, we have provided a detailed and unified response in the Global Rebuttal. Here, we would like to briefly clarify that in Table 1, the reason CAMP does not appear to outperform all other methods is not due to CAMP's failure to learn a good representation. Rather, it is because we included the reward-based method IQL as a baseline. In the single-task setting, CAMP learns solely from preference data, while IQL learns from actual rewards, making it more challenging for CAMP to learn an optimal policy compared to IQL. However, when comparing CAMP with other preference-based methods like PT and OPPO, we find that CAMP still demonstrates significant advantages.

3. Regarding the multi-task dataset case, Tables 4 and 5 indicate that CAMP performs well on some tasks but poorly on others. Could you explain why this is the case? Is it due to an imbalance in the training of the multi-task scenario?

Thank you for your question. Tables 4 and 5 show the performance of different methods on the MT-10 tasks with near-optimal and sub-optimal data. The inconsistency in CAMP's performance across the 10 tasks—where it performs well on some tasks but poorly on others—is not due to an imbalance in multi-task training. Instead, it reflects the inherent difficulty of the tasks. Some tasks within the MT-10 set are more challenging, while others are relatively easier. This variability in task difficulty explains the differences in performance observed. Additionally, many tasks in MetaWorld have the characteristic that even if the cumulative reward reaches 90% of the maximum reward, the success rate remains 0. This also contributes to the large variance observed in the results presented in Tables 4 and 5.

We hope that our responses above, as well as those in the global rebuttal, have addressed your concerns. If they have, we kindly ask you to consider improving our score. If not, we welcome further discussion.

Dear Reviewer Y24x,

We first would like to thank the reviewer's efforts and time in reviewing our work. We were wondering if our responses have resolved your concerns. Since the discussion period is ending soon, we would like to express our sincere gratitude if you could check our reply to your comments. We will be happy to have further discussions with you if there are still some remaining questions! We sincerely look forward to your kind reply!

Best regards,

The authors

Thanks for taking the time to provide feedback on our paper. We appreciate your valuable comments and would like to address each of your concerns.

1. It is recommended that the authors emphasize these distinctions and update the contribution list.

Thanks for your suggestion. We will emphasize the distinctions between CAMP, OPPO, and MTDiff in the main text and update the contribution list in the next version. Specifically, our contributions can be summarized as follows:

• We have introduced multi-task preferences and proposed to learn high-dimensional preference representations from trajectory segments, which extends the versatility of traditional preference-based methods like OPPO.
• We have proposed a mutual information regularized diffusion model for multi-task trajectory generation by conditioning on preference representations, saving the need for actual rewards and expert prompts used in MTDiff.
• We have conducted extensive experiments on MetaWorld, showing comparable performance to reward-based methods, superiority to baseline preference-based methods, and favorable generalization performance to unseen tasks.

2. The proposed method does not achieve the best performance across all metrics. It is suggested that the authors provide a corresponding discussion.

Thanks for your question. We found that this issue is quite similar to the second question raised by Reviewer Y24x. Therefore, we have provided a detailed and unified response in the Global Rebuttal. Please refer to it for more information. Here, we can briefly summarize some conclusions: preference data is not as accurate and abundant as reward data. Hence, methods that learn from preference data are less likely to perform as well as reward-based methods. Our experiments in both multi-task and single-task settings aim to demonstrate that CAMP, using only preference data, can achieve performance comparable to the best reward-based methods, while significantly outperforming other preference-based methods.

3. It is advised to include a discussion on the computational costs of the model, such as space complexity and time complexity.

Thanks for your insightful question. Regarding the concerns of the reviewer about the optimization of mutual information and diffusion models, we have proposed corresponding improvement methods. These include scaling and simplifying the original mutual information objective, as well as employing faster sampling methods for diffusion models. Please refer to our response to Question 2 from Reviewer L2wb for more details, where we have discussed which parts of our approach were more time-consuming and how CAMP was optimized to address these issues.

Here, we have conducted a further comparison of the space complexity and time complexity of CAMP, with the diffusion model-based multitask learning method, MTDiff [1]. Both methods were run on an Nvidia RTX 4090 GPU.

According to the above results, the runtime for each update in MTDiff is approximately half of that in CAMP. This is because, in addition to optimizing the diffusion model, CAMP also optimizes the preference representation network and the optimal preference representation. It is worth noting that the learning of preference representation and the optimization of the diffusion model can be decoupled; if decoupled, our method might exhibit greater flexibility and efficiency. On the other hand, by comparing the GPU memory usage during algorithm execution, we found that the space required by CAMP is about one-third of that required by MTDiff. This indicates that, although CAMP is more computationally intensive, its memory space requirements are significantly lower. This reason may be that CAMP utilizes a U-Net based noise predictor, while MTDiff needs a more complex Transformer-based noise predictor.

[1] He H, Bai C, Xu K, et al. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. NeurIPS 2023

4. It would be beneficial to discuss the impact of the quality of the provided preference data on the results.

Thanks for your suggestion. We would like to discuss the impact of the quality of preference data from two perspectives: noisy preferences and the quality of provided trajectory data. Firstly, noisy preferences can indeed influence the accuracy of learned preference representations. To make representation learning robust to noisy preferences, our method has incorporated two useful components: 1) inspired by contrastive learning methods [1], our method also constructs positive and negative samples and optimizes the representation network so that they can be separated in the representation space. This is more robust than learning a reward model aligned with preferences in traditional preference-based RL methods. 2) Our method learns a representation distribution, which captures the uncertainty contained in the preference dataset [2].

Secondly, in multi-task learning experiments, we tested learning results using near-optimal and sub-optimal datasets, which vary in the quality of trajectory data. The near-optimal datasets, which contain more expert trajectories, have higher quality compared to sub-optimal datasets. Figure 4 illustrates that our method is robust to different qualities of trajectory data.

[1] Shen W, Zhang X, Yao Y, et al. Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards. ArXiv:2403.07708.

[2] Duan S, Yi X, Zhang P, et al. Negating negatives: Alignment without human positive samples via distributional dispreference optimization. ArXiv:2403.03419.

We hope that our responses above, as well as those in the global rebuttal, have addressed your concerns. If they have, we kindly ask you to consider improving our score. If not, we welcome further discussion.

Dear Reviewer 3Xmu,

We first would like to thank the reviewer's efforts and time in reviewing our work. We were wondering if our responses have resolved your concerns. Since the discussion period is ending soon, we would like to express our sincere gratitude if you could check our reply to your comments. We will be happy to have further discussions with you if there are still some remaining questions! We sincerely look forward to your kind reply!

Best regards,

The authors

Thanks for taking the time to provide feedback on our paper. We appreciate your valuable comments and would like to address each of your concerns.

1. Table 1 presents a large variation in performance for different tasks with the same implementation method. What factors contribute to this phenomenon, and how do we choose the optimal approach for a given task?

Thanks for your questions. The significant variation in performance for the same method across different tasks, as seen in Table 1, is primarily due to the varying quality of the offline data used during the learning process. Specifically, the single-task performance is evaluated on the commonly used D4RL Gym Mujoco tasks for offline RL, which include halfcheetah, walker2d, and hopper tasks. We employed three offline datasets: medium, medium-replay, and expert, which are obtained by different behavior policies. The trajectory quality in the medium and medium-replay datasets is lower than that in the expert dataset. Therefore, each method's performance on the medium and medium-replay datasets is generally lower than on the expert dataset.

As for how to choose the optimal approach for a given task, it depends on which training data are given. According to Table 1, for single-task learning, CAMP is less effective than IQL which utilizes actual rewards. This is understandable because reward signals provide more detailed and accurate information compared to preference data, and learning with actual rewards in single tasks can potentially yield better performance. Despite this, only given preference data, CAMP outperforms other preference-based methods. Therefore, if the given task provides actual reward signals, reward-based methods are generally superior. However, if the given task only has preference data, CAMP is better than other preference-based methods.

2. The proposed method seems to have high computational complexity.

Thanks for your question. Regarding computation complexity, we have presented specific results for space complexity and time complexity in our response to Question 3 from Reviewer 3Xmu. Please refer to that for details. Here, we analyze the parts of the method that are more time-consuming and introduce how we addressed these issues. The computation in our method primarily involves two parts:

• Training the MI Regularized Diffusion Model: Estimating the mutual information (MI) between the generated trajectory $\tau_0$ and the preference representation $w$ is computationally intensive, as generating $\tau_0$ requires $k$-step denoising. To mitigate this, we apply approximations and replace $I(\tau_0;w)$ with $I(\tau_k;w)$, thus bypassing the denosing process during training. $I(\tau_k;w)$ can be obtained from the estimated noise at each step, requiring only a two-layer MLP to align the dimensions with $w$.
• Inference Stage: Sampling during the inference stage is time-consuming, which we have discussed in Appendix F. To address this, we have experimented with DPM-solver [1] to reduce the sampling time of the diffusion model . This improvement has boosted the inference speed by a factor of 8.4 compared to the previous method. Specifically, on an Nvidia RTX3090, the denoising time per step has been reduced from 15.2ms to 1.8ms. Despite the improvement in efficiency, our initial results indicate a compromise in the quality of the generated trajectories, likely due to the complexity of generating continuous trajectories. We will explore further enhancements to balance efficiency and quality in our future work.

[1] Lu C, Zhou Y, Bao F, et al. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS 2022

3. Whether the proposed method receive a greater impact on efficiency as the type and number of tasks increase?

Thanks for your question. This issue is quite similar to the first question raised by Reviewer Y24x, so we have included it in the Global Rebuttal. More experimental results can be found in our response to Question 1 from Reviewer Y24x. Here, we provide a brief conclusion: As the types and number of tasks increase, the learning of preference representations improves, but it becomes more challenging for the diffusion model to fit the trajectory distributions of multiple tasks. The final performance is influenced by both of these factors.

4. From Table 2, the average performance on the generalization ability for unseen tasks is still not very good.

Thanks for your question. The average performance for unseen tasks is not very good due to the inherent challenge of generalization in MetaWorld, where the difference between the learned and unseen tasks is substantial. This discrepancy causes the preference representations of new tasks learned by CAMP to be distant from those of existing tasks in the representation space. Although we incorporate MI regularization to enable the diffusion model to generate trajectories corresponding to the task preferences and exhibit generalization in the representation space, this generalization ability is weakened when the preference representations of new tasks differ significantly from those of existing tasks. This leads to the performance in Table 2 not being particularly outstanding. Nevertheless, CAMP still demonstrates superior performance compared to existing methods.

5. Is the generalization ability of this approach feasible in practical applications?

Thanks for your question regarding the feasibility of the generalization ability of this approach in practical applications. While the current results indicate that the generalization ability is not yet at an optimal level, there is potential for improvement. With sufficient high-quality data and a diverse set of tasks, the generalization ability of the model can be further enhanced, making it more feasible for practical applications.

Dear Reviewer L2wb,

We first would like to thank the reviewer's efforts and time in reviewing our work. We were wondering if our responses have resolved your concerns. Since the discussion period is ending soon, we would like to express our sincere gratitude if you could check our reply to your comments. We will be happy to have further discussions with you if there are still some remaining questions! We sincerely look forward to your kind reply!

Best regards,

The authors