RTDiff: Reverse Trajectory Synthesis via Diffusion for Offline Reinforcement Learning

Thanks for your insightful and inspiring comments! We provide the following clarifications in response to your concerns.

1. W1: Comparison with ROMI

• We thank the reviewer for highlighting this related work. We have updated the paper to include a comparison of this work in Appendix C.5. We want to emphasize that our approach is significantly different from ROMI. First, ROMI focuses on learning the environment's dynamics and employing a reverse rollout strategy based on reverse dynamics. In contrast, our method directly generates trajectories using diffusion models, which avoids the inaccuracies inherent in dynamics learning and rollout strategies, resulting in higher-quality generated trajectories.
• Additionally, our work introduces a novel strategy to fully leverage the diffusion model's ability to generate reliable long-length trajectories. Specifically, we incorporate an OOD detector to control the generation length. This detector ensures that the generated trajectories extend beyond the offline data distribution, introducing new environmental information to enhance offline RL algorithms. Simultaneously, it prevents trajectories from straying too far from the distribution, maintaining reliability and utility.
• Furthermore, we develop a noise control strategy that can be applied to any diffusion-based data augmentation method. This technique is particularly effective when the number of samples is small and can be utilized with any diffusion-based data augmentation approach for RL.
• Overall, our work introduces several distinct techniques compared to ROMI. To empirically demonstrate the advantages of RTDiff, we include ROMI as a baseline in our experiments. The results clearly show that RTDiff outperforms ROMI. | | IQL+RTDiff| IQL+ROMI| TD3BC+RTDiff| TD3BC+ROMI| | ------ | ------------------ | ---- | ---- | ---- | |maze2d-umaze| 8.3 |5.4|10.2|9.6| |maze2d-medium | 3.3 |2.1|9.8|9.4| |maze2d-large | 14.3 | 8.1|7.7|3.5|

2. W2: Visualization of the experiments

• Thanks for the valuable comments. To better illustrate the generated trajectories in real environments used in the experiments, we have presented a new visualization of generated trajectories in the Maze2D environment in Appendix D of the updated version of the paper.

3. W3: Typos

• Thanks for pointing this out. We have fixed this issue in the updated version of the paper.

4. Analysis of visual environments

• First, we would like to clarify that we have already included the experimental results in the visual reinforcement learning environments in Section 5.3 of our original paper.
• Here we provide an additional analysis in the Meta-world environment, similar to that presented in Section 6.. We choose Coffee Push as the example task. The results show that even in the visual environments, RTDiff still generates fewer risky transitions and more useful transitions.

3. W3: Fidelity of the generated trajectories Thanks for your valuable comments. We have added a section in Appendix C.6 of the updated version of the paper to include the quantitative evaluation of generated trajectories. Below we show the fidelity of the generated trajectories. To measure the fidelity of the generated samples, we follow the previous works using two statistics: Marginal: Mean Kolmogorov-Smirnov [Ref1] and Correlation: Mean Correlation Similarity [Ref2]. As expected, the results show that RTDiff does not aim to generate more realistic trajectories, but rather to produce more diverse samples that lie outside the distribution, thereby benefiting the RL performance. This is because RTDiff generates adaptive, longer trajectories compared with other baselines, attributed to our proposed OOD detector and reverse synthesis model. Therefore, we would like to further emphasize that, while we agree with the reviewer that fidelity is an important factor in assessing data generation in general, our focus here is more on the "usefulness" of the generated data, specifically how it improves RL performance.

[Ref1] The kolmogorov-smirnov test for goodness of fit. Frank J. Massey Jr. 1951.

[Ref2] Tests for rank correlation coefficients. E. C. Fieller, et al. 1957.

Thanks for your insightful and inspiring comments! We provide the following clarifications in response to your concerns.

1. W1: Comparison with ROMI

• We thank the reviewer for highlighting this related work. We have updated the paper to include a comparison of this work in Appendix C.5. We want to emphasize that our approach is significantly different from ROMI. First, ROMI focuses on learning the environment's dynamics and employing a reverse rollout strategy based on reverse dynamics. In contrast, our method directly generates trajectories using diffusion models, which avoids the inaccuracies inherent in dynamics learning and rollout strategies, resulting in higher-quality generated trajectories.
• Additionally, our work introduces a novel strategy to fully leverage the diffusion model's ability to generate reliable long-length trajectories. Specifically, we incorporate an OOD detector to control the generation length. This detector ensures that the generated trajectories extend beyond the offline data distribution, introducing new environmental information to enhance offline RL algorithms. Simultaneously, it prevents trajectories from straying too far from the distribution, maintaining reliability and utility.
• Furthermore, we develop a noise control strategy that can be applied to any diffusion-based data augmentation method. This technique is particularly effective when the number of samples is small and can be utilized with any diffusion-based data augmentation approach for RL.
• Overall, our work introduces several distinct techniques compared to ROMI. To empirically demonstrate the advantages of RTDiff, we include ROMI as a baseline in our experiments. The results clearly show that RTDiff outperforms ROMI. | | IQL+RTDiff| IQL+ROMI| TD3BC+RTDiff| TD3BC+ROMI| | ------ | ------------------ | ---- | ---- | ---- | |maze2d-umaze| 8.3 |5.4|10.2|9.6| |maze2d-medium | 3.3 |2.1|9.8|9.4| |maze2d-large | 14.3 | 8.1|7.7|3.5|

2. W2: Explanation of the idea

• We want to clarify that the core idea of this work is not to directly reduce the ratio of generated data that falls outside the offline data distribution but rather to mitigate the issues caused by such trajectories. The problem with data distribution arises from the overestimation of transitions in the fake trajectories generated during data augmentation. These overestimated transitions can directly impact the strategies learned by the agents, leading to degraded performance.
• Our approach addresses this issue by performing synthesis in reverse order. This ensures that even if some trajectories deviate from the offline data distribution, the overestimation of such transitions does not directly harm performance.
• As analyzed in Section 6, we categorize transitions into four types, where only the In2In transitions are strictly within the offline data distribution. Among the other three categories, the In2Out transitions are particularly detrimental to performance. Our goal is not to eliminate the other three categories entirely or ensure that all generated transitions are in-distribution. Instead, we focus on reducing the number of In2Out transitions.
• As demonstrated in Table 7, reverse synthesis can even result in fewer In2In transitions, which are less useful for data augmentation. This finding further validates the effectiveness of our approach.

Thanks for the authors' detailed reply. I have no further questions and I'm willing to raise my score.

4. Q3: Quantitative evaluation of generated trajectories

• Thanks for your valuable comments. We have included this in Appendix C.6 of the updated version of the paper, where we present the quantitative evaluation of generated trajectories.
• Below we show the average model error of the generated trajectories. To measure the model error of the generated samples, we calculate the normalized error between the synthesized states and the real states after transition, which is defined as $(T(s,a) - s')^2$ for a transition $(s, a, s')$. As expected, the results show that RTDiff does not aim to generate more realistic trajectories, but rather to produce more diverse samples that lie outside the distribution, thereby benefiting the RL performance. This is because RTDiff generates adaptive, longer trajectories compared with other baselines, attributed to our proposed OOD detector and reverse synthesis model.
• Therefore, we would like to further emphasize that, while we agree that the quality is an important factor in assessing data generation, our focus here is more on the "usefulness" of the generated data, specifically how it improves RL performance. | | RTDiff | SynthER | ATraDiff | | ------ | ------------------ | ---- | ---- | |maze2d-umaze| 0.05 | 0.02 | 0.03 | |maze2d-medium | 0.06 | 0.03 | 0.03 | |maze2d-large | 0.11 | 0.07 | 0.08 |

Thanks for your insightful and inspiring comments! We provide the following clarifications in response to your concerns.

In particular, we discuss in detail how our approach significantly differs from ROMI and model-based RL methods, providing empirical comparisons that validate our approach's superior performance. Additionally, we would like to emphasize that the use of diffusion models for data augmentation in RL is a relatively emerging area. Prior works such as SynthER and ATraDiff also focus on using diffusion models for data augmentation, with their novelty stemming from the specific techniques used to exploit diffusion models. Within this context, we believe the novelty of our work should not be diminished solely because we are not the first to use diffusion for data augmentation.

1. W1/Q1: Comparison with ROMI

• We thank the reviewer for highlighting this related work. We have updated the paper to include a comparison of this work in Appendix C.5. We want to emphasize that our approach is significantly different from ROMI. First, ROMI focuses on learning the environment's dynamics and employing a reverse rollout strategy based on reverse dynamics. In contrast, our method directly generates trajectories using diffusion models, which avoids the inaccuracies inherent in dynamics learning and rollout strategies, resulting in higher-quality generated trajectories.
• Additionally, our work introduces a novel strategy to fully leverage the diffusion model's ability to generate reliable long-length trajectories. Specifically, we incorporate an OOD detector to control the generation length. This detector ensures that the generated trajectories extend beyond the offline data distribution, introducing new environmental information to enhance offline RL algorithms. Simultaneously, it prevents trajectories from straying too far from the distribution, maintaining reliability and utility.
• Furthermore, we develop a noise control strategy that can be applied to any diffusion-based data augmentation method. This technique is particularly effective when the number of samples is small and can be utilized with any diffusion-based data augmentation approach for RL.
• Overall, our work introduces several distinct techniques compared to ROMI. To empirically demonstrate the advantages of RTDiff, we include ROMI as a baseline in our experiments. The results clearly show that RTDiff outperforms ROMI. | | IQL+RTDiff| IQL+ROMI| TD3BC+RTDiff| TD3BC+ROMI| | ------ | ------------------ | ---- | ---- | ---- | |maze2d-umaze| 8.3 |5.4|10.2|9.6| |maze2d-medium | 3.3 |2.1|9.8|9.4| |maze2d-large | 14.3 | 8.1|7.7|3.5|

2. W2: Comparisons with model-based RL

• We thank the reviewer for highlighting that model-based RL methods can also serve as strategies for offline data augmentation. However, the generation quality of model-based methods heavily depends on both the accuracy of the learned dynamics and the rolling-out strategy used to produce trajectories. These factors can make it challenging to achieve reliable data augmentation for RL. It is worth noting that ROMI is an example of a model-based data augmentation method.
• Our approach, in contrast, incorporates several novel components specifically designed to improve the quality of generated data, such as the OOD detector and noise control mechanisms.
• To provide an empirical comparison between our method and model-based RL approaches, we have included an additional experiment in Appendix C.5 of the updated version of the paper, comparing our method to MOPO as suggested by the reviewer. The results below demonstrate that our method outperforms model-based methods in terms of performance, further highlighting its effectiveness. | | IQL+RTDiff| IQL+MOPO| TD3BC+RTDiff| TD3BC+MOPO| | ------ | ------------------ | ---- | ---- | ---- | |maze2d-umaze| 8.3 | 5.1 | 10.2 | 9.6 | |maze2d-medium | 3.3 | 1.7 | 9.8 | 8.8 | |maze2d-large | 14.3 | 5.9 | 7.7 | 2.6 |

3. Q2: RL methods hyperparameters

• After the data augmentation, we did not adjust any hyperparameters of the RL methods.

5. Theoretical contributions

• This paper primarily focuses on algorithmic development and empirical evaluation, rather than on theoretical analysis or formal proofs. Similar to previous works like ROMI, SynthER, and ATraDiff, our objective is to propose practical methodologies and validate their effectiveness through comprehensive experimental results. We leave the exploration of theoretical guarantees and formal proofs as interesting future work.

Thanks for your insightful and inspiring comments! We provide the following clarifications in response to your concerns:

1. Incorporating OOD detector with other baselines

• Thanks for the valuable question. We would like to clarify that the OOD detector is particularly useful for our reverse synthesis model, where forward synthesis methods cannot benefit from the OOD detector.
• Specifically, in our method, the OOD detector is used to control the length of the generated trajectories. This is crucial because we aim to generate trajectories that extend beyond the offline data distribution to provide new information to the agent, while ensuring they do not deviate too far, which could reduce their usefulness and increase risk. The effectiveness of our OOD detector is demonstrated in Tables 4 and 12. Our results show that the OOD detector outperforms any fixed-length generation strategy. Additionally, we observed that setting the threshold too high or too low negatively impacts performance.
• To better understand why reverse synthesis is useful, we want to further show that this OOD detector is not useful for other synthesis methods. First of all, SynthER only performs transition-level synthesis, which inherently does not need to control the generation length. Also we want to emphasize that with forward synthesis like ATraDiff, generating trajectories going out of the offline data distribution is more risky, as transitions going from inside to outside may lead to performance degradation. To validate this, we show the results of combining ATraDiff with an OOD detector of different thresholds. From the results below, we found that those forward synthesis methods derive limited benefits from the OOD detector, which validated the unique effectiveness of reverse synthesis in our framework. | | RTDiff (Ours) | ATraDiff | ATraDiff with dis_m = 1.0 | ATraDiff with dis_m = 1.3 | ATraDiff with dis_m = 1.5 | ATraDiff with dis_m = 2.0 | --- | ----- | ---- | ---- | ---- | ---- | ---- | |maze2d-umaze|12.3 | 7.1 | 7.4 | 7.0 | 6.3 | 6.0 | |maze2d-medium |8.3 |6.2 | 6.6 | 6.3 | 5.6 | 5.4 | |maze2d-large | 11.3 | 7.8 | 7.7 | 7.8 | 7.4 | 7.1|

2. Incorporating noise control with other baselines

• Our proposed noise control is largely independent of the specific generation method and can be applied to any data augmentation approach using a diffusion-based framework. It demonstrates particularly effective when the number of examples is limited. Notably, this technique is also applicable to SynthER and ATraDiff, as demonstrated by the results shown below. We use SynthER and ATraDiff to generate both 1M data with and without the random generation. The results show that the noise control can consistently improve the performance. | | SynthER with Noise control| SynthER with random generation | ATraDiff with Noise control | ATraDiff with random generation | | ------ | ------------------ | ---- | ---- | ---- | | maze2d-umaze | 2.3 | 1.7 | 3.1 | 2.8 | | maze2d-medium | 0.6 | 0.3| 1.6 | 1.2 | |maze2d-large | 7.3 | 6.3 | 8.1 | 5.2 |

3. Explanation of illustration examples

• Thanks for the valuable comment. We first sincerely apologize for any confusion and would like to clarify that the reviewer may have misunderstood our illustration example. The results presented in Table 7 were generated in the Maze2D-large environment, not in the environment depicted in Figure 2.
• The trajectories depicted in Figure 2 are intended to intuitively illustrate the behavioral differences between reverse synthesis and normal forward synthesis, highlighting the benefits of reverse synthesis. These trajectories are purely for illustrative purposes and were not generated by the models.
• To better showcase the real generated trajectories in the environments investigated in this paper, we have included a new visualization of the generated trajectories in the Maze2D environment in Appendix D of the updated version of the paper.

4. Why normal synthesis with OOD detector has more In2Out transitions?

• Thanks for the question. This is because the generation length in normal synthesis without an OOD detector is set to be a fixed length, which must be kept shorter than the average generation length with an OOD detector to avoid performance decrease from generating risky trajectories. In contrast, forward synthesis with an OOD detector allows for an increased average generation length, because the OOD detector can avoid generating too long trajectories. Therefore, forward synthesis with an OOD detector can generate some trajectories different from the offline data distribution. As demonstrated by the results in Table 2, forward synthesis with an OOD detector can also produce more transitions of Out2Out and significantly fewer In2In transitions which might be redundant.