ContraDiff: Planning Towards High Return States via Contrastive Learning

Q5. Did you experiment with setting different thresholds for what qualifies as “low” or “high” return? If so, how did varying these thresholds impact the learned policy or the model’s ability to generalize to novel tasks?
R5. Thank you for your question. \ (1) We have conducted the experiments of thresholds for positive and negative samples (i.e., the thresholds for what qualifies as “low” or “high” return) and hyperparameters (ξ, ζ, σ) in Appendix A.11. In summary, analysis of the hyper-parameters indicates that their impact on model performance is moderate, and searching for the optimal hyper-parameters is easy. Please refer to Appendix A.11 for more detailed discussions. (2) The choice of threshold on novel tasks can be made a priori based on the state return in the dataset. For example, one can visualize the state-return distribution of the dataset, and take the return with the most dramatic return change as the threshold. For example, thresholds can be identified as turning points by calculating the rate of change (e.g., differences) or the second-order rate of change (acceleration)[1,2]. After that, we can fix the threshold and adjust $\sigma$ for better results. \

[1] Satopaa, V., Albrecht, J., Irwin, D., & Raghavan, B. (2011, June). Finding a" kneedle" in a haystack: Detecting knee points in system behavior. In 2011 31st international conference on distributed computing systems workshops (pp. 166-171). IEEE.
[2] Silverman, B. W. (2018). Density estimation for statistics and data analysis. Routledge.

Q1. While the paper presents contrastive learning as a means to exploit low-return data, it could better clarify why this mechanism is theoretically optimal for avoiding low-return states compared to other potential approaches, such as weighting adjustments or imitation-based filtering.
R1. Thank you for your comment. RL methods based on weighting adjustments or imitation-based filtering usually assign weights to states or trajectories according to their returns, aiming to prioritize high-return samples during the optimization process and ignore low-return samples. In other words, these methods do not actively attempt to distance the model from low-return samples. In contrast, our approach uses contrastive learning, treating low-return samples as negative examples and high-return samples as positive examples. This not only brings generated trajectories closer to high-return trajectories but also actively pushes them away from low-return samples. The comparison with the reweighting methods (summarized in Table 2) demonstrates the effectiveness of our proposed approach. We also designed ablation studies in Section 4.3.1 to demonstrate the effectiveness of utilizing low-return states.

Q2. The authors are not clear enough in describing the use of weighted contrast loss to constrain trajectory generation.
R2. Thank you for your comment. The weighted contrast loss is formulated in Eq.(13), which is further used for contractive learning, as formulated in Eq.(14). Details of Eq.(13) and Eq.(14) are discussed in Section 3.3. Specifically, at time step $t_1$, the agent obtains the observation from the environment and generates a future trajectory of $H$ steps, $[ (s_h, a_h) ]_{t \leq h \leq h+H}$. Considering that the importance of predictions decreases as the time step extends further into the future, we apply a weighting scheme to the contrastive learning loss for the next $H$ steps. For the contrastive learning loss at time step $h$, we assign a weight of $1/(h+1)$.

Q3. In the 3.3 model learning section, I noticed that the optimized trajectory generation by minimizing the Mean Square Error between the ground truth and neat trajectory predicted. Therefore, the diffusion should denoise the data from the noisy data completely. I think it's more expensive to train like this. Can you show some experiments comparing the cost of training?
R3. Thank you for your comment. The cost of training with the Mean Square Error between the ground truth and neat trajectory predicted (i.e., Eq.(11)) is the same as predicting the noise. Following EDP [1] and the official repo of Diffuser [2], we adopt the one-step denoising method to directly predict the original trajectory within one step rather than denoising the data from the noisy data completely, for additional supervisory information during the training phase. However, during the testing phase, we still generate trajectories (i.e., future plans) through the complete reverse denoising process.

[1] Kang, B., Ma, X., Du, C., Pang, T., & Yan, S. (2024). Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36.
[2] Janner, M., Du, Y., Tenenbaum, J. B., & Levine, S. (2022). Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991.

Q4. ContraDiff planning towards high return states, leading policy improvements. It is very similar to some offline RL methods, such as SAW[1], A2PR[2], LAPO[3]. Can you add some discussion with these methods in the related works or more experiments comparison? R4. Thank you for your suggestion! We have added discussions of SAW[1], A2PR[2], and LAPO[3] to enrich the Related Works section. We ported SAW to Diffuser (Diffuser-SAW) for experimentation. Since A2PR is not applicable to Diffuser, we conducted experiments with A2PR on our suboptimal datasets. We were unable to perform experiments related to LAPO because the code for LAPO was not provided in the paper. The comparison of ContraDiff with Diffuser-SAW and LAPO on Walker2d-Rand-Exp is shown in the table below. It can be observed that ContraDiff outperforms Diffuser-SAW and A2PR in all cases, demonstrating the advantage of ContraDiff.

Dear Reviewer Ku5d,

We have provided detailed responses to your concerns many days ago, we hope these responses have adequately addressed your concerns. As the discussion phase is coming to an end, we sincerely request your further responses.

If we have resolved your issues, please consider raising your score to the positive side. If you have any further questions, please feel free to share them with us! Any valuable feedback is crucial for improving our work.

Best regards,
The Authors

Dear Reviewer Ku5d,

We have provided detailed responses to your concerns days ago, and have provided the performance of ContraDiff without the expert dataset accordingly. We hope our responses have adequately addressed your additional concern. As the discussion phase is coming to an end, we sincerely request your further responses.

If we have resolved your issue, please consider raising your score to the positive side. If you have any further questions, please feel free to share with us! We would appreciate your further discussion on our paper.

Best regards, The Authors

Dear Reviewer Ku5d,

Thank you for your effort in reviewing our paper! We are glad that we have addressed most of you concerns. We have provided the performance of ContraDiff without expert data accordingly several days ago, and we hope our responses have adequately addressed your additional concern. As the discussion phase is coming to an end, we sincerely request your further responses.

If we have resolved your issue, please consider raising your score to the positive side. If you have any further questions, please feel free to share with us! We would appreciate your further discussion on our paper.

Best regards,
The Authors

Q1. The relationship between contrastive loss and RL loss needs further detailed justification. Intuitively, these two losses appear highly correlated, especially when considering low-return trajectories as negative samples. A numerical analysis comparing these losses across trajectories with different returns would be valuable to verify if they indeed show similar trends. If so, the contrastive loss might merely serve as an enhancement to the RL loss rather than a meaningful regularization term that needs to be balanced against the primary objective.
R1. Thank you for your comments.
(1) The contrastive loss and RL loss are independent. The contrastive loss (i.e., Eq.(13) and Eq.(9)) is derived from InfoNCE, aiming to increase the expected return of generated trajectories by maximizing the distance from low-return trajectories while minimizing the distance to high-return trajectories. This effectively pulls the distribution of generated trajectories toward high-return trajectories and pushes it away from low-return trajectories. In contrast, the RL loss (i.e., Eq.(11)) simply attempts to fit the distribution of trajectories across the entire dataset.
(2) Nevertheless, following your suggestion, we visualized the magnitude of the contrastive loss with and without contrastive learning on Walker2d-Rand-Exp-0.1, and presented this visualization in Appendix A.14 in our revised paper. Note that:

and

The closer the distance to high-return states and the farther the distance to low-return states, the smaller the value of Eq. (13). It can be observed that, regardless of the use of contrastive learning, the RL loss shows a downward trend and eventually stabilizes. However, for the value of Eq.(13), when contrastive learning is not used, the contrastive loss remains large and fluctuates significantly. On the other hand, when contrastive learning is applied, the contrastive loss decreases. Moreover, ContraDiff shows a lower value on RL Loss + Eq.(13) than ContrasDiff w/o CL. This demonstrates that (1) the RL loss and contrastive loss are independent, with the contrastive loss serving as a meaningful regularization term; and (2) contrastive learning indeed maximizes the distance from low-return trajectories while minimizing the distance to high-return trajectories, as further corroborated by the analyses in Sections 4.3.

Q2. The methodological differences illustrated in Figure 1(d)(e)(f) lack sufficient justification.This figure fails to demonstrate why pushing away from low-return states is fundamentally superior to upweighting high-return trajectories, or why return-based contrastive learning outperforms traditional trajectory-based approaches.
R2. Thank you for your comment. As shown in Figure 1 (d), up-weighting high-return trajectories can only achieve the high-return trajectory ($s$ -> $G$) that exists within the dataset. However, when the agent is trapped in low-return states, e.g., $s_t$ in Figure 1 (d), there is no corresponding high-return example in the dataset (e.g., $s_t$ -> $G$), and up-weighting high-return trajectories fails. Our method, as shown in Figure 1 (e), by constraining the future states, takes the state from the high-return trajectory ($s$ -> $G$) as the target state at $s_t$, thereby enabling a transition from $s_t$ to a high-return trajectory. Besides, there are also some other methods that use contrastive learning in reinforcement learning (Figure (f)). However, they focus on learning better representations. For example, they aim to minimize the similarity between representations of adjacent states within the same trajectory while maximizing the distance between representations of states from different trajectories, as illustrated in Figure 1(f).

Q3. The paper's method of determining positive and negative samples relies heavily on hyperparameters (ξ, ζ, σ) without theoretical justification for their value ranges. The paper lacks a principled approach to determine these thresholds across different environments. A more systematic study on how different return thresholds affect the distribution of positive/negative samples and their impact on learning dynamics would strengthen the method's foundation.
R3. Thank you for your comment.
(1) The main idea of our paper is to introduce contrastive learning to enhance decision-making. Therefore, we focus on evaluating the effectiveness of the idea instead of justifying the hyperparameters. As shown in Table 1 and Table 2, even using the most rudimentary positive and negative sample selection method (i.e., dividing solely based on thresholds), ContraDiff still demonstrates significant advantages in most scenarios.
As for the choice of threshold, a priori can be made based on the state return in the dataset. For example, one can visualize the state-return distribution of the dataset, and take the return with the most dramatic return change as the threshold. For example, thresholds can be identified as turning points by calculating the rate of change (e.g., differences) or the second-order rate of change (acceleration)[1,2]. After that, we can fix the threshold and adjust $\sigma$ for better results.
(2) We have conducted the experiments of thresholds for positive and negative samples and hyperparameters (ξ, ζ, σ)  in Appendix A.11.  In summary,  analysis of the hyper-parameters indicates that their impact on model performance is moderate, and searching for the optimal hyper-parameters is easy. Please refer to Appendix A.11 for more detailed discussions.
[1] Satopaa, V., Albrecht, J., Irwin, D., & Raghavan, B. (2011, June). Finding a" kneedle" in a haystack: Detecting knee points in system behavior. In 2011 31st international conference on distributed computing systems workshops (pp. 166-171). IEEE.
[2] Silverman, B. W. (2018). Density estimation for statistics and data analysis. Routledge.\

Q4. The current state-level contrastive learning approach breaks the temporal correlation between states within trajectories, as it samples positive and negative states purely based on return values (SR) or clustering results (SRD). This design might lose important sequential information that could be better captured through trajectory-level contrastive learning. I encourage authors to explore trajectory-aligned state sampling strategies and conduct comparative experiments between state-level and trajectory-level contrastive learning to provide more solid empirical evidence for the design choices.
R4. Thank you for your comment and suggestion. (1) We believe by using "the important sequential information", you are referring to the dynamic consistency. We want to clarify that we designed the cluster-based approach (i.e., SRD) to ensure dynamic consistency, as is described in Section 3.2.1. Specifically, for a state $s_t$, we first find the cluster it belongs to, marked as $C_t$. Next, we treat all the next states of states in $C_t$ as the positive candidates of $s_t$. As all the states in $C_t$ are grouped into the same cluster as $s_t$, we consider these states to be similar, and thus the next state of any in-cluster state of $C_t$ is also considered reachable from $s_t$. Thus, selecting the positive sample of $s_t$ from the next states of in-cluster states ensures the temporal consistency of the constructed contrastive samples.
(2) Nevertheless, following your suggestion, we conducted experiments on trajectory-level contrastive learning (denoted as $\text{ContraDiff}^{\text{t}}$ ) on Walker2d-Rand-Exp, and the results are as follows. As can be observed, ContraDiff outperforms $\text{ContraDiff}^{\text{t}}$ in all scenarios, indicating that our state-level contrastive learning method does not significantly affect the sequential information.

Q5. The competitive performance of baselines like CDE and HD raises questions about the necessity of the proposed contrastive mechanism, especially considering these methods achieve comparable results without any specific designs for handling low-return trajectories. While ContraDiff shows improvements in certain scenarios, the performance overlap with these simpler methods suggests that the advantages of explicitly handling low-return trajectories might be less significant than claimed. This observation calls for a more thorough analysis of what unique benefits, if any, are brought by the contrastive mechanism for low-return trajectory utilization.
R5. Thank you for your comment.
（1）Table 1 presents the comparison between ContraDiff and several recent SOTA baselines on suboptimal datasets, where ContraDiff achieves the best or second-best performance on 25 out of 27 datasets. This preliminarily demonstrates the advantage of ContraDiff in effectively leveraging negative samples.
（2）Furthermore, to better showcase the advantages of ContraDiff, we compare it with SOTA methods designed to handle imbalanced data. We initialize the environment with random low-return states to create more challenging scenarios, aiming to evaluate the ability of ContraDiff and SOTA methods to handle situations where the agent is trapped in low-return states. Specifically, we investigate the capability of ContraDiff to transition from low-return states to high-return states. As is shown in Table 2, ContraDiff achieves the best performance on almost all (23 out of 27) datasets compared to other SOTA baselines. Since the difference between ContraDiff and the baseline lies in leveraging negative sample information effectively through contrastive learning, results in Table 2 further highlight that utilizing low-return samples can effectively help the model escape from low-return areas.
（3）To provide a more intuitive explanation, we visualize in Figure 3 the ability of ContraDiff, AW+Diffuser, and their baseline, Diffuser, to escape from low-return states. As shown, when facing similar low-return states, AW+Diffuser achieves better results through its reweighting mechanism but still fails to fully escape the low-return region (terminating after 40 steps). In contrast, with the help of contrastive learning, ContraDiff effectively leverages negative sample information and maintains a healthy trajectory even after 60 steps, successfully escaping the low-return state. This visualization intuitively demonstrates the advantage of our proposed contrastive learning framework in fully utilizing negative sample information.

Dear Reviewer 2zbW,

We have provided detailed responses to your concerns many days ago, we hope these responses have adequately addressed your concerns. As the discussion phase is coming to an end, we sincerely request your further responses.

If we have resolved your issues, please consider raising your score to the positive side. If you have any further questions, please feel free to share them with us! Any valuable feedback is crucial for improving our work.

Best regards,
The Authors

Thank you for the authors' rebuttal. However, I regret that my fundamental concerns remain unaddressed:

1. The authors' characterization of RL loss as 'simply fitting trajectory distribution' is fundamentally incorrect and misleading. This description fits vanilla behavioral cloning, not RL losses, which are specifically designed to learn from high-return trajectories through value estimation and advantage weighting. The claimed independence between contrastive and RL losses is therefore questionable, as both inherently bias towards high-return trajectories.

2. The response merely differentiates the losses mathematically without demonstrating their fundamental independence or complementarity rather than redundancy.

3. Most critically, the authors fail to acknowledge or compare their work with 'Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline RL', which presents a more theoretically complete treatment of combining contrastive learning with diffusion models in offline RL.

Given these unresolved issues - the unclear relationship between contrastive and RL losses, insufficient comparison with critical baseline methods, and inadequate positioning relative to important related work - I believe this work does not yet meet the bar for publication.

Dear reviewer,
We greatly appreciate you letting us know that some of your concerns have not been fully addressed. Also, we found that the formulas in our previous response (R1) were not rendered correctly. We sincerely apologize for any inconvenience this may have caused you. We have fixed the issue, and the formulas are now properly rendered. Here are our further responses:

Q1. The unclear relationship between contrastive and RL losses. The claimed independence between contrastive and RL losses is therefore questionable, as both inherently bias towards high-return trajectories.
R1. Thank you for your further comment.
Firstly, we want to clarify that the RL loss mentioned in our previous reply refers to the loss used in Diffusion-based methods such as Diffuser. In the works based on Diffusion for decision-making, such as Diffuser, they apply an MSE loss between real data and generated trajectories to train the Diffusion model. Only during evaluation do they adopt value-guided sampling to generate trajectories with higher returns. Therefore, the learned diffusion model simply fits the trajectory distribution.
However, the contrastive loss used in our paper is designed to better leverage the information embedded in negative samples (i.e., low-return trajectories). By maximizing the distance between generated trajectories and negative samples while minimizing the distance between generated trajectories and positive samples(i.e., high-return trajectories), the contrastive loss constrains the generated trajectories to a high-return area.

Secondly, even methods based on the Actor-Critic framework, such as IQL and CQL, their loss basically assign weights to different ($s$, $a$) pairs based on the advantage of $a$, which fails in actively utilizing the information from low-return samples, for example, they do not actively distance the generated samples from low-return trajectories. In contrast, by employing a contrastive loss, our model can fully leverage the information from low-return trajectories, and avoid low-return samples. The comparisons in Table 2 demonstrate the advantages of our proposed contrastive loss over these reweighting methods. \

Furthermore, the difference between the contrastive loss and the RL loss can be visually demonstrated through the experiments. Figure 5 and Figure 6 in Section 4.3 of our paper intuitively demonstrate the advantages of our contrastive loss. In Figure 5, we can observe that compared to without using contrastive loss (Figure 5(b)), employing contrastive loss (Figure 5(c)) results in more high-reward states, as indicated by the warmer-colored sample points in the figure. To further illustrate the advantages of the contrastive loss, Figure 6 presents the reward distributions when value guidance is not applied, comparing cases with and without contrastive loss. As shown, without contrastive loss (orange region), the rewards obtained by the model closely align with the reward distribution of the dataset itself (green region). However, when using contrastive loss (blue region), the reward distribution of the model is highly concentrated in the high-reward region, with minimal presence in the low-reward region. Figures 5 and 6 intuitively demonstrate the benefits brought by the proposed contrastive learning loss.

Q2. Insufficient comparison with critical baseline methods.
R2. Thank you for your further comment.  
First of all, we want to clarify that, as mentioned in the Introduction, we focus on addressing the data imbalance problem by making better use of low-return samples in the dataset. Therefore, it is more fair to compare our method with other approaches aimed at solving data imbalance.

As is shown in Table 2, we compare ContraDiff with data imbalance works that were recently published on ICLR 2023 and NIPS 2023, including advantage-weighting (AW) [1], return-weighting (RW) [1],  density-ratio weighting with advantage (AW-DW) [2] and density-ratio weighting with uniform (U-DW) [2]. The results show that ContraDiff achieves optimal or sub-optimal results in 25 out of 27 situations.  Besides, we have compared ContraDiff with plenty of regular SOTA RL methods in Table 1, in which ContraDiff achieves optimal or sub-optimal results in 25 out of 27 situations.

[1] Hong, Zhang-Wei, et al. "Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting." The Eleventh International Conference on Learning Representations.
[2] Hong, Zhang-Wei, et al. "Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets." Advances in Neural Information Processing Systems 36 (2023): 4985-5009.

Dear Reviewer 2zbW,

Thank you for your effort in reviewing our paper! We have provided detailed responses to your concerns several days ago, and we hope our responses have adequately addressed your additional concerns. As the discussion phase is coming to an end, we sincerely request your further responses.

If we have resolved your concerns, please consider raising your score to the positive side. If you have any further questions, please feel free to share with us! We would appreciate your further discussion on our paper.

Best regards,
The Authors

Dear Reviewer 2zbW,
Thank you for your further comment.
Q1. About the understanding of RL loss in diffusion-based methods.
R1. The diffusion model-based methods like Diffuser [1] are typically optimized with two losses: the diffusion loss and the guidance loss (i.e., the guidance you have mentioned). The diffusion loss is usually defined as MSE. For example, in Diffuser, the diffusion model loss is defined as (in page 4 of diffuser paper [1]):

The guidance loss is used to predict the value of each action, which can be observed in the official repo of Diffuser. We agree with your opinion that previous methods adopt value-guided sampling can generate better actions (or trajectories). However, the capability of value-guided sampling is actually learned by the guidance loss instead of the diffusion loss. As can be seen from the formula, return is not utilized in the diffusion loss, and therefore the diffusion loss essentially just simply fits the trajectory distribution.

The effect of diffusion losses can be validated in our experimental results. In Figure 6, we visualize the probability density functions of the rewards obtained by ContraDiff and ContraDiff-C when no guidance is used, where ContraDiff-C denotes the setting of removing the contrastive learning loss from ContraDiff. It is evident that the probability density obtained by ContraDiff-C is closer to the distribution of the dataset than ContraDiff, indicating the diffusion loss essentially just simply fits the trajectory distribution. Moreover, ContraDiff is more concentrated in the high-reward region, indicating that our proposed contrastive learning indeed actively moves away from low-return samples, and moves toward the high-return samples.

[1] Janner, Michael, et al. "Planning with diffusion for flexible behavior synthesis." arXiv preprint arXiv:2205.09991 (2022).

Q2. Traditional actor-critic methods like IQL and CQL inherently achieve a similar "pushing away" effect that our contrastive learning aims to accomplish.
R2. Generally, the RL loss of advantage-based methods adopts a form similar to IQL, where the RL loss of IQL is as follows (Eq.7 from the IQL paper [1]）:

As can be seen, for samples with low advantage, the weight is closer to 0, This means that when a sample has a very low return, these methods choose to neglect the information in those samples. Hence, essentially speaking, these methods mainly focus on learning from higher-return samples (i.e., how to achieve a high return), and traditional actor-critic methods like IQL and CQL do not exhibit the 'pushing away' effect we mentioned. Ablation studies in Section 4.3.1 show that ignoring the information from low-return samples leads to poor performance in most cases.

In contrast, our method learns from both low-return and high-return samples (i.e., how to achieve a high return and how to avoid the low-return), making our method able to actively avoid the low-return area (our unique advantage). As shown in Figure 3, we initialize ContraDiff and the baseline methods in a low-return state to explore their ability to escape from the low-return state. As can be observed, even though AW, a method focused on learning from high-return samples, takes a few more steps than Diffuser, it eventually receives terminal signals. In contrast, ContraDiff goes further and ultimately maintains a healthy posture to keep moving forward. In other words, with the contrastive loss, ContraDiff makes it better to escape from the low-return region compared to methods based on advantage-weighting.

Overall, The key difference between our method and previous works is previous methods focus on learning from high-return samples, but our method learns from both high and low-return samples.

[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021).

Q3. The difference between our method ContraDiff and QGPO [1].
R3. We are glad that we have reached a consensus on this point that, as we mentioned in R3 of our last response, our method and QGPO utilize contrastive learning in different ways and are applied to solve different problems.

Nevertheless, following your suggestion, we conducted experiments of QGPO on Walker2d-Rand-Exp, and the results are as follows. As can be observed, ContraDiff outperforms QGPO in 2 out of 3 settings, indicating that ContraDiff, leveraging contrastive learning to pull the generated samples closer to high-return samples and actively push them away from low-return samples, achieves better results compared to QGPO, which uses contrastive learning to learn higher guidance. We will also add the relevant experimental results to our paper.

As we had included a section in Related Works (Section 5.3) that provides a detailed overview of methods using contrastive learning in RL, given the different approaches and objectives in utilizing contrastive learning, we have given an appropriate introduction to QGPO in Section 5.3.

[1] Lu, Cheng, et al. "Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

Q1. While the idea of using contrastive learning is well motivated, the section on how "positive" and "negative" examples are identified requires additional explanations. The paper did not specify how the clustering is performed -- if the clustering is to enforce "dynamic consistency", I would assume the information is already available in the offline dataset? For me it is unclear how well the clustering captures reachability or state transitions beyond just grouping next states together. This could be a limitation in environments with complex or non-linear dynamics.
R1. Thank you for your comment. We believe there is a misunderstanding regarding the details of clustering.
(1) We simply group all the states in the dataset into clusters. Specifically, for a state $s_t$, we first find the cluster it belongs to, marked as $C_t$. Next, we treat all the next states of states in $C_t$ as the positive candidates of $s_t$. As all the states in $C_t$ are grouped into the same cluster as $s_t$, we consider these states to be similar, and thus the next state of any in-cluster state of $C_t$ is also considered reachable from $s_t$. Thus, selecting the positive sample of $s_t$ from the next states of in-cluster states ensures the temporal consistency of the constructed contrastive samples. Note that all the required info is available in the dataset.
(2) However, as we understand it, you are referring to treating the same-cluster states of $s_t$'s next state as $s_t$'s next state. This represents a fundamental difference in exploration ability: In ContraDiff, the next states of same-cluster states are not necessarily within the same cluster, which allows ContraDiff to explore a broader range of potential next states. However, if we treat the same-cluster states of $s_t$'s next state as $s_t$'s next state since these states are all within the same cluster, they are inherently similar, which limits ContraDiff's exploration capacity. Overall, selecting contrastive samples from the next states of $s_t$'s same-cluster states is not the same as selecting contrastive samples from the same-cluster states of $s_t$'s next state. The next states of same-cluster states used in ContraDiff provide more diverse contrastive learning samples than the same-cluster states of $s_t$'s next state, and as a result, ContraDiff has greater exploration ability.
(3) We believe you are referring to scenarios where the environment dynamics may change over time. For these cases, we can use sub-segments of the trajectories as clustering units, and take into account both $s_t$​ and its historical information when searching for reachable future states in the training phase. Since historical information contains dynamics of the environment, clustering that incorporates historical information can effectively overcome the challenges posed by complex or non-linear dynamics. Exploration in non-linear dynamic scenarios is a promising direction, and we will consider it as part of our future work.

Q2. The paper lacks discussion on the complexity or runtime implications introduced by k-means clustering, especially for large offline RL datasets.
R2. Thank you for your comment. Clustering is a preprocessing step, performed only once and stored in a file. During training, we only need to look up the clustering results, so clustering will not impact the training speed of the model. Moreover, the need for clustering information exists only during the training phase and is not required during the testing phase.
Nevertheless, we measured the time spent on clustering across different datasets, and the results are shown in the table below. Please note that we have taken several measures to optimize the clustering process. For example, we set the number of clusters to the square root of the number of states, and configure appropriate clustering batch sizes. It can be observed that even during the preprocessing stage, the clustering operation does not consume significant time.

Q5. It is unclear how the baseline results were obtained -- whether they came from existing code, were reimplemented by the authors, or taken from other sources.
R5. Thank you for your comment. The codes of other baselines were all obtained from their official repo. For the results of baselines in Table 1 and Table 2, we obtain the results by running the existing code of their official repo with default hyper-parameters. The results of baseline methods in Table 3 are adopted from their own papers and Diffuser paper.

Q6. The writing, especially in the experiments section, is not very clear to understand. Some sentences and typos make the paper harder to follow.
R6. Thank you for your comment! We have revised our paper based on your suggestions. Please refer to our revised paper for details.

Q7. When comparing Table 3 with Table 1, it seems like introducing the expert trajectory leads to a decrease in performance, unexpectedly. For instance, DT achieves a score of 36.6 at original HalfCheetah-MR dataset but 7.5, 6.7, 6.1 in three conditions where expert data is introduced. I was wondering if the same codebase is used in achieving these results, and if so, could the authors share insights on why DT’s performance drops so drastically with the added expert data?
R7. Thank you for your comment. We have rechecked the code we used, and we confirm that the code we are using is from DT's official repo (as long as the other baselines). Note that since the DT paper demonstrated that the target return-to-go value has minimal impact on the model, we initially selected the maximum value from the dataset as the target return-to-go (i.e., RTG) for better performance. We pulled the code again from the official repository of DT and conducted relevant experiments, with results consistent with those reported in our paper.

Still, we experimented with different return-to-go values on Halfcheetah-Rand-Exp, and the results are shown in the table below. It can be observed that excessively large target return-to-go values caused a performance decline. When the target return-to-go was reduced to 6000, we achieved comparable results. As DT achieves a score of 36.6 at the original HalfCheetah-MR dataset on the same setting, it can be concluded that expert data helps in improving DT's performance. Even though, ContraDiff still demonstrates a significantly better trend compared to DT.

Q8. In most tasks ContraDiff-SR outperforms ContraDiff-SRD -- this seems unexpected given the author's intuition mentioned in section 3 that ContraDiff-SR may ignore the transition dynamics in its sampling process?
R8. Thank you for your question. Ensuring temporal consistency is not always appropriate in all cases. In some environments and datasets, enforcing temporal consistency based on existing data may limit the model's exploration of states with higher returns. Although these states are not explicitly indicated as reachable in the dataset, they can be reached through the generalization capability of diffusion models, making ContraDiff-SR show better performance. Details of the scenarios where ensuring temporal consistency based on existing data can yield better results are discussed in Appendix A.4, please refer to our paper for details.

Q9. How would the proposed model perform on sparse-reward tasks like AntMaze? This benchmark is commonly used in offline RL evaluations but results for it are missing here.
R9. Thank you for your suggestion! Following Diffuser and other baselines, we primarily evaluate ContraDiff's performance on locomotion, Maze2D, and Kitchen environments, in which Maze2D and Kitchen are sparse-reward tasks. Nevertheless, we have conducted experiments on Antmaze datasets, and the results are as follows. As can be observed, although Antmaze datasets are reward-diverse, ContraDiff outperforms Diffuser in most situations.

Q3. The increases in performance are mostly marginal. For many tasks the increase falls within one standard deviation of the baseline methods.
R3. Thank you for your comment. In Section 4, we demonstrate the unique advantages of ContraDiff from multiple perspectives. Specifically,
（1）Table 1 presents the comparison between ContraDiff and several recent SOTA baselines on suboptimal datasets, where ContraDiff achieves the best or second-best performance on 25 out of 27 datasets. Even after accounting for the potential impact of variance, ContraDiff delivers outstanding results. For example, on HalfCheetah-Rand-Exp-0.1, ContraDiff outperforms the second-best method by 28.1%. This preliminarily demonstrates the advantage of ContraDiff in effectively leveraging negative samples.
（2）Furthermore, to better showcase the advantages of ContraDiff, we compare it with SOTA methods designed to handle imbalanced data. We initialize the environment with random low-return states to create more challenging scenarios, aiming to evaluate the ability of ContraDiff and SOTA methods to handle situations where the agent is trapped in low-return states. Specifically, we investigate the capability of ContraDiff to transition from low-return states to high-return states. As is shown in Table 2, ContraDiff achieves the best performance on almost all (23 out of 27) datasets compared to other SOTA baselines. For example, ContraDiff outperforms the second-best method by 41.4% on Hopper-Rand-Exp-0.1. Since the difference between ContraDiff and the baseline lies in leveraging negative sample information effectively through contrastive learning, results in Table 2 further highlight that utilizing low-return samples can effectively help the model escape from low-return areas.
（3）To provide a more intuitive explanation, we visualize in Figure 3 the ability of ContraDiff, AW+Diffuser, and their baseline, Diffuser, to escape from low-return states. As shown, when facing similar low-return states, AW+Diffuser achieves better results through its reweighting mechanism but still fails to fully escape the low-return region (terminating after 40 steps). In contrast, with the help of contrastive learning, ContraDiff effectively leverages negative sample information and maintains a healthy trajectory even after 60 steps, successfully escaping the low-return state. This visualization intuitively demonstrates the advantage of our proposed contrastive learning framework in fully utilizing negative sample information.

Q4. The authors mentioned "comparing ContraDiff with other regular methods" but didn't include specific indication of where the results are presented. I later found the results in the appendix, but the authors may consider additional illustrations and explanations on the results. Furthermore methods like CQL are only tested on the standard D4RL datasets but not the mixture dataset proposed by the authors. More experiment results would be needed here to better demonstrate model performance.
R4. Thank you for your comment. （1）We have revised the description based on your comment to avoid potential misunderstandings. We summarize the results comparing ContraDiff with newly proposed SOTA methods in Table 1. However, due to the paper length limitation, we present the results compared with traditional methods in Table 3. The relevant analysis for comparisons with newly proposed SOTA methods is in Section 4.2.1, while the analysis for comparison with diffusion-free methods is in Appendix A.3.
(2) Nevertheless, based on your suggestion, we evaluated the performance of CQL, IQL, and MOPO on HalfCheetah-Rand-Exp, Walker2d-Rand-Exp, and Hopper-Rand-Exp, and compared them with ContraDiff. The results are shown in the table below. As can be observed, ContraDiff achieves the best performance in almost all scenarios.

Dear Reviewer DdY7,

We have provided detailed responses to your concerns many days ago, we hope these responses have adequately addressed your concerns. As the discussion phase is coming to an end, we sincerely request your further responses.

If we have resolved your issues, please consider raising your score to the positive side. If you have any further questions, please feel free to share them with us! Any valuable feedback is crucial for improving our work.

Best regards,
The Authors