DyDiff: Long-Horizon Rollout via Dynamics Diffusion for Offline Reinforcement Learning

• Q1: How does the accuracy of the single-step dynamics model affect the final performance? How would sub-optimal initial policies and dynamics models impact the result? Are there some visualizations of how the rollouts evolve across different iterations? I recommend showing some visualization results to show how the iterative process improves the rollouts. It is hard to tell if the improvement comes from the iteration or other components with current results.

• A1: Thanks for your valuable question. We visualize the synthetic trajectories in Maze2D tasks with different accuracy of the single-step dynamics model and the learning policy, as well as how they evolve across different iterations. Please refer to Appendix E.6 for the visualization results and detailed analysis.

• Q2: In Figure 3a, the method seems sensitive to the iteration time $M$. How could one determine a good value by observing the dataset and the environment?

• A2: Emperically, the choice of $M$ depends on the complexity of the environment. For example, we choose $M=2$ for MuJoCo locomotion tasks, while $M=1$ for Maze2D tasks with relatively simple dynamics. In practice, we find that using larger $M$ will not necessarily decrease the performance but definitely cost more training time. So searching $M=1,2$ is sufficient for most cases.

• Q3: In section 5.3, why does DyDiff improve much more than locomotion tasks? Does the improvement come from sparse rewards or easier dynamics in the environments?

• A3: We would like to contribute this to the modeling ability of long-horizon distributions. As the visualization results in Appendix E.6 show, both simple single-step MLP models and DyDiff can capture the dynamics in Maze2D tasks. However, the single-step model can hardly realize the obstacles, providing many trajectories going through the walls. These trajectories are correct in dynamics if we omit the wall, but they obviously do harm to the policy learning. In contrast, DyDiff can model the long-horizon distribution using diffusion models and synthesize most legal trajectories. This comparison proves the improvement comes from not only the easier dynamics but also the modeling ability of DyDiff.

• Q4: What is the training resource of DyDiff compared with baselines in terms of time and model sizes?

• A4: Thanks for your advice. We have provided the training time and the model sizes of DyDiff and SynthER in Appendix E.7. As an add-on method, DyDiff does not affect the inference time of the underlying policies.

• Q1: Preliminaries are sufficient for a reader familiar with the field but might be brief for readers unfamiliar with either Diffusion Models or RL.

• A1: Thanks for your advice. We have added a section to provide detailed preliminaries in Appendix B.

• Q2: The notation in Section 4, especially the trajectory superscript, confused me. It appears the authors use the superscript both for the diffusion step i and “trajectory improvement iteration” k. If I conclude correctly, the superscript is in parenthesis if it regards the trajectory improvement iteration. However, this is not consistent, e.g. from line 274 on in section 4.2. Perhaps a brief introduction to the notation, e.g. in section 4.1, and more consistency could help.

• A2: We apologize for possible confusion caused by our notations. There are two types of timesteps in Diffusion Models for RL, i.e., the timestep of states and actions in RL trajectories and the timestep in the diffusion process. To represent the iteration times of DyDiff, we have to use parenthesis superscripts. We add a brief introduction in Section 4.2, and fixed confused notations from Line 274, marked in red.

• Q3: I would have liked to see a bit more on what this paper contributes in comparison to concurrent work of Jackson et al.

• A3: The work PGD[1] identifies a similar policy mismatch problem in applying diffusion synthetic data to RL policy training. It computes the log-likelihood of synthetic trajectories related to the learning policy, which is then used as guidance in the denoising process. In contrast, DyDiff addresses this problem using a different approach without guidance, thus can work even though the log-likelihood is intractable. For example, TD3BC trains a deterministic policy, so PGD has to re-model it as a unit Gaussian distribution to obtain the log-likelihood, while DyDiff can be applied to it seaminglessly. Besides, we develop theoretical analysis about the generation ability of general DMs. 

• Q4: The results in Section 5.2 indicate that DyDiff improves the performance in some settings, decreases performance in other settings and seems almost identical in the rest of settings. Without any significance tests the claim that “DyDiff significantly improves the performance of these algorithms without any additional hyperparameter tuning.” seems over the top. Please provide significance tests and/or rephrase the claim to be more precise. Similar to the results in 5.2, the ablation studies in 5.4 seem to have little reliability due to high variance.

• A4: Thanks for your advice, and we have removed the word "significantly". Also, we compute the probability of improvement (PI) following RLiable[2] for our main experiments. Note that DyDiff is an additional method to underlying model-free policies, so we only compare DyDiff+X with its counterpart X. The results are listed below.

• Q5: In line 232 you state that “Though we can now use DMs to generate state trajectories, the initial action trajectory for the condition is still left blank.” while Figure 2 already told the reader that the initial action sequence is initialized with the actions from the policy during one-step rollouts. For me the lines 182 to 186 thus caused a lot of confusion, which was then resolved later when reading section 4.2. Is there a particular reason for this structure or could it be adapted? Rather a suggestion: The caption of Figure 2, and the corresponding description in lines 181-186 could be more clear.
• A5: We apologize for confusion caused by our ambiguous description in Figure 2, Line 181-186, and the last paragraph of Section 4.1. In our original writing, although we have introduced the main structure of the algorithm, we still write as if the reader did not know how the initial action sequence is obtained. We have revised the text in the caption of Figure 2, Line 181-186, and the last paragraph of Section 4.1, marked in red.

• Q6: In Section 5.2 you state that “From the perspective of different base policies, DyDiff exhibits relative incompatibility with CQL”, which seems intuitive, but nonetheless, CQL seems to have similar benefit from DyDiff than the other baselines in Section 5.3. Do you have an intuition on this?
• A6: We would like to contribute this improvement to the modeling ability on long-horizon distributions, which is much more important in sparse-reward tasks than dense-reward ones. In Maze2D, the agent is not rewarded until it reaches the goal. DyDiff can synthesize high-quality long-horizon rollouts that can help the policy find the path to the goal quickly. Therefore, even CQL can benefit from DyDiff in these tasks.

[1] Jackson, Matthew Thomas, et al. "Policy-guided diffusion." arXiv preprint arXiv:2404.06356 (2024).

[2] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." NeurIPS 2021.

• Q1: Also, it is unclear why just replacing the generated actions with the action of the learning policy, the mismatch issue can be addressed as the learning policy is also obtained from the real training dataset.

• A1: We would like to first clarify our motivation and the policy mismatching problem. This problem arises when the synthetic data do not follow the distribution induced by the learning policy. Although the learning policy is also obtained from the real dataset, it is continuously updated, and its induced distribution keeps changing. By using the actions of the learning policy, we explicitly enforce the generated trajectory distribution to stay closer to the changing distribution in each iteration. In our motivating example in Figure 1, we have shown that augmenting data following the same distribution to the policy will promote more on policy learning, which is further supported by the improvements in our main experiments.

• Q2: In theorem 1, the authors claimed that over long horizons, the single-step error bound \epsilon_m is even less than the accumulative multi-step error bound \epsilon_d as DMs has superior modeling capabilities, and then concluded that DMs are better for long-horizon rollout than single-step dynamics model. However, no theoretical guarantees are provided.

• A2: We apologize that our statements may not be so clear that misunderstanding arises. Both $\epsilon_m$ and $\epsilon_d$ are conditions assumed to be already known. Because they model the errors related to complicated neural networks that can hardly be analyzed theoretically, we can only evaluate their emperical values. In fact, our theoretical analysis follows a common approach in model-based reinforcement learning used in many previous works [1,2,3], and they only provide empirical error bounds as well. In the uploaded revision, we have cleared our statement in Section 4.3, marked in red.

• Q3: In the proposed approach, states and actions are first generated simultaneously, then the generated actions are replaced with the real ones. Compared with DecisonDiffuser where only state sequences are generated, what are the advantages of the proposed method?

• A3: Thanks for your valuable question. We motivate differently from DecisionDiffuser in that we aim to build a dynamics model allowing interaction with other policies. In contrast, DecisionDiffuser binds the dynamics model and the policy, so it cannot interact with arbitrary policies. It fills the actions via a pre-trained inverse dynamics model, which does not introduce any new information to the generated sequence. As for DyDiff, the pre-trained diffusion model acts like a dynamics model. Though it first generates states and actions simultaneously, the action sequence is then replaced by actions given by the learning policy itself. Such replacement is the interaction between the diffusion model and the policy, outputing synthetic data that follow both the dynamics and the distribution induced by the policy.

• Q4: In which conditions, can Assumptions 1 and 2 be satisfied?

• A4: We apologize for the confusion caused by our unclear explanation. In fact, Assumption 1 can be decomposed into the following two assumtions:

  - Assumption 1.1: $\max_t D_\mathrm{TV}(T_d(s_t|s_0,\tau_a) \| T(s_t|s_0, \tau_a)) \le \epsilon_{s,d}$

  - Assumption 1.2: $\max_t D_\mathrm{TV}(T_d(s_t|s_0, \tau_{a,d}) \| T_d(s_t|s_0, \tau_{a})) \le C_{a,d}\max_t \|\tau_{a,d} - \tau_a\|$

  Using the triangular inequality of $D_\mathrm{TV}$, we can obtain Assumption 1. Now, assumption 1.1 is the same as the condition in Theorem 1. 

  For Assumptions 1.2 and 2, they are similar to the Lipschitz condition. Assumption 1.2 bounds the change in the state distribution when the action sequence changes, and Assumption 2 bounds the change in the action distribution when the state changes. In practice, when input states and actions do not fall far from the data coverage of the training set, these assumptions can be assumed to hold. In the far-out-of-distribution region, the accuracy of models becomes too low for us to predict their behavior.   In the uploaded revision, we have added this explanation and the detailed derivation in Appendix D.4. 

• Q5: In the experimental results, only the training time of DyDiff is reported. The training time of other baselines should also be stated and compared with the training time of DyDiff.
• A5: Thanks for your advice. We have updated the training time of other baselines in Appendix E.7 and included this as a limitation in Section 6. Also, we would like to point out that the training time in offline RL is usually less important than that in online RL.

[1] Janner, Michael, et al. "When to trust your model: Model-based policy optimization." NeurIPS 2019.

[2] Lai, Hang, et al. "Bidirectional model-based policy optimization." ICML, 2020.

[3] Yu, Tianhe, et al. "Mopo: Model-based offline policy optimization." NeurIPS 2020.

Thank you for your careful review and insightful feedback on our paper. We have made every effort to address your comments, including:

• Our motivation for replacing the generated actions with the actions derived from the learning policy.
• A detailed explanation of the theoretical analysis included in the revised paper.
• The key differences between DyDiff and DecisionDiffuser.
• The training times of other baselines.

If you have any further questions or concerns regarding our paper or responses, please do not hesitate to let us know. We look forward to engaging in constructive discussions to further improve our paper.

• Q1: In such a case, how can we claim that \epsilon_m is less than \epsilon_d? If this assumption cannot be satisfied, does Dydiff still have theoretical advantages over single-step models?
• A1: We would like to clarify the purpose of our theoretical analysis. The analysis aims to demonstrate that if

using non-autoregressive generative models, such as Diffusion Models, is preferable to single-step models. In our experiments, we find that condition $(∗)$ holds for the Diffusion Model employed in DyDiff, which we believe benefits from generating all timesteps simultaneously, thereby avoiding compounding errors. However, the condition $(\*)$ can only be determined emperically. For example, consider a Diffusion Model without training—it would generate random samples for which $(\*)$ does not hold, and it would clearly not outperform single-step models. Thus, we do not claim that Diffusion Models inherently have theoretical advantages over single-step models. Instead, under empirical conditions, they are very likely to outperform single-step models.

• Q2: In Dydiff, high-reward trajectories are selected to improve the quality of the training data by pre-training a reward model. Is such a filtering mechanism also adopted for other data synthesizers ( e.g., SynthER) in the experiments? That only the high-quality synthesis data is used for the data augmentation.
• A2: Thank you for your question, and we apologize for the lack of detailed analysis regarding our filtering mechanism. Although the original SynthER does not include a filtering scheme, following your suggestion, we applied the softmax filter to the data generated by SynthER, sampling transitions based on the softmax of their rewards. We then trained TD3BC agents on the filtered datasets, with the results provided in Appendix E.2 of our revised paper. The results indicate that directly picking out high-reward transitions leads to a slight improvement in performance compared to the original SynthER. However, the performance gains are primarily observed in relatively simple tasks, such as Hopper and Walker2d. Conversely, filtered SynthER underperforms relative to the original SynthER on more complex tasks like HalfCheetah and Maze2d-large. This may occur because selecting high-reward transitions limits the training data to better but less accessible regions, which does not necessarily benefit policy learning. For DyDiff, we apply the filtering scheme at the trajectory level, preserving the complete paths leading to high-reward regions.

Thank you again for raising such valuable questions, which have helped us further improve the paper. We hope the above responses address your concerns. If they do, we would greatly appreciate it if you could consider increasing the paper's rating.

Thank you for your question regarding Theorem 1, which is now the key disagreement between us. We would like to further clarify our response as follows:

• We sincerely apologize for any confusion caused by the typo in our previous response. The factor should be expressed as $\frac{\gamma}{1-\gamma} \cdot \frac{\epsilon_m}{\epsilon_d}$, consistent with the notation in the paper. This value is greater than $1$ if $\epsilon_d < \frac{\gamma}{1-\gamma} \epsilon_m$, and specifically, $\epsilon_d < 99 \epsilon_m$ when $\gamma=0.99$. In other words, a DM achieves a better return gap bound than a single-step model as long as its error is less than $\frac{\gamma}{1-\gamma}$ times that of the single-step model. With this corrected formulation, this condition aligns with intuition and much easier to satisfy compared to the incorrect version, especially given the superior modeling capabilities of DMs.

• For the value of $\gamma$, we provide a partial list of commonly used RL algorithms, including both online and offline RL methods, where $\gamma=0.99$ is a standard setting.

  • Online RL:
    • DQN[1], Extended Data Table 1
    • DDPG[2], Section 7
    • TRPO[3], Table 2, 3
    • PPO[4], Table 3, 4, 5
    • SAC[5], Table 1
    • TD3[6], Table 3
    • MBPO[7], in its official implementation
  • Offline RL:
    • CQL[9], Section F
    • IQL[10], in its official impelementation
    • TD3BC[11], Table 4
    • MOPO[12], in its official impelementation
    • DiffQL[13], in its official impelementation

  Additionally, most RL algorithms set $\gamma > 0.9$, with very few works using $\gamma < 0.5$. This is because $\gamma < 0.5$ results in highly myopic policies, where long-horizon rollouts are unnecessary.

• Regarding the theoretical validity of Theorem 1, we would like to clarify that incorporating parameters requiring empirical determination is a common practice in theoretical analyses of MBRL[7][12][14]. For instance, in the widely used MBPO[7], Theorem 4.3 includes $\epsilon_\pi$ and $\epsilon_{m'}$, which are left for empirical determination. As our paper adopts a similar framework to MBPO, such parameters also need to be empirically determined.

  In MBPO, the theoretical analysis concludes with the following statement:

  While this insight does not immediately suggest an algorithm design by itself, we can build on this idea to develop a method that makes limited use of truncated, but nonzero-length, model rollouts.

  Following this reasoning, we do not believe the inclusion of empirical parameters undermines the validity of Theorem 1. Instead, our theoretical analysis is intended to provide insights into why DMs are advantageous, rather than to offer a definitive guarantee that DMs will always outperform other approaches.

[1] Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." nature 518.7540 (2015): 529-533.

[2] Lillicrap, T. P. "Continuous control with deep reinforcement learning." arXiv preprint arXiv:1509.02971 (2015).

[3] Schulman, John. "Trust Region Policy Optimization." arXiv preprint arXiv:1502.05477 (2015).

[4] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).

[5] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018.

[6] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." International conference on machine learning. PMLR, 2018.

[7] Janner, Michael, et al. "When to trust your model: Model-based policy optimization." Advances in neural information processing systems 32 (2019).

[8] Hafner, Danijar, et al. "Mastering diverse domains through world models." arXiv preprint arXiv:2301.04104 (2023).

[9] Kumar, Aviral, et al. "Conservative q-learning for offline reinforcement learning." Advances in Neural Information Processing Systems 33 (2020): 1179-1191.

[10] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021).

[11] Fujimoto, Scott, and Shixiang Shane Gu. "A minimalist approach to offline reinforcement learning." Advances in neural information processing systems 34 (2021): 20132-20145.

[12] Yu, Tianhe, et al. "Mopo: Model-based offline policy optimization." Advances in Neural Information Processing Systems 33 (2020): 14129-14142.

[13] Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. "Diffusion policies as an expressive policy class for offline reinforcement learning." arXiv preprint arXiv:2208.06193 (2022).

[14] Lai, Hang, et al. "Bidirectional model-based policy optimization." International Conference on Machine Learning. PMLR, 2020.

Dear Reviewer bgMC,

Thank you for your effort and time on reviewing our manuscript. As today marks the final day of the review period, we would like to kindly ask if there are any remaining questions or concerns that we can address to assist in your evaluation. We greatly appreciate your time and feedback and remain available to provide any clarifications you may need.

Best regards,

Authors of Submission5681

• Q6: How about directly comparing with some diffusion-based planning algorithms using the learning policy as guidance?
• A6: It is hard to directly compare DyDiff with diffusion-based planning algorithms as they do not possess a decoupled policy. Specifically, diffusion-based planning methods, or diffusion planners, usually directly plan long-horizon state-action trajectories, or state trajectories but fill the actions by an inverse dynamics model. In both cases, they determine the action to be executed by the planned trajectories without the help of independent policies. Therefore, they do not use the learning policy as guidance, which is different from DyDiff in principle.

• Q1: Could the authors provide further analyses and experiments to discuss this problem, including providing a more detailed analysis on how the state distributions and the dynamics consistency change during the iteration?

• A1: Thanks for your question, where we may not describe the refinement process of DyDiff very clearly. We would like to clarify this process again. In the initial trajectory, each action $a_t$ is generated by the learning policy $\pi$ given $s_t$, so the trajectory is consistent with $\pi$. The problem now is $(s_t, a_t, s_{t+1})$ may not follow the real dynamics as the single-step dynamics model is not accurate. After the DM takes $s_0$ and $\tau_a$ as conditions to refine this trajectory, the newly generated $(s_t', a_t, s_{t+1}')$ obeys the real dynamics. However, $a_t$ does not necessarily follow the distribution $\pi(s_t')$. Therefore, the generation process of DM does not remain the policy to be the learning policy, and we further refine this trajectory iteratively with the learning policy and the DM. As an illustrative example, we calculate the total MSE error of the generated trajectorie with a TD3BC policy in the hopper-medium task and plot how it changes with the iteration times in Appendix E.5. 

• Q2: How do you get the learning policy $\pi$ in the generation process? Does it update iteratively with the learning process? If I understand correctly, does it mean that you first initialize the learning policy then generate trajectories with DyDiff and learn a new policy based on the generated trajectories? Also, what is the frequency of policy updates in relation to the trajectory generation process, and how this will affect the overall performance?

• A2: Yes, the learning policy is first initialized and then continuously updated by the underlying offline RL algorithm. In fact, we "update" the policy with both true and generated trajectories rather than learn a "new" policy. As for the frequency of policy updates, lowering this frequency is equivalent to increasing the generated data amount in each generation process. We conduct experiments on hopper-medium-v2 with TD3BC, finding this hardly affects the performance, as listed in the following table, because the proportion of synthetic data used for updating the policy is controlled by the real ratio $\alpha$. We generate 2000 trajectories in the main paper and never tuned this hyperparameter.

• Q3: Why DyDiff can significantly improve the performance on Maze2d compared with Mojuco? Can the authors provide a more detailed analysis between the results on these two environments?

• A3: We contribute the significant improvement of DyDiff on Maze2d to the ability of modeling long-horizon distributions, so DyDiff can synthesize high-quality long trajectories. We visualize the synthetic trajectories in Maze2d, as shown in Appendix E.6. In Maze2d tasks, the agent does not receive any reward until it reaches the goal, so effective exploration will accelerate the learning process and lead the learning policy to find a shorter path to the goal. On the contrary, MuJoCo tasks provide dense rewards, so the benefit from exploration is smaller than on Maze2d tasks.

• Q4: I'm wondering if DyDiff can still improve the performance with some other baselines like Decision Transformer.

• A4: Thanks for your advice about applying DyDiff on Decision Transformer (DT). However, the structure of DyDiff must be modified to be compatible with the return-to-go used in DT, and training DT costs a lot of time. Our experiments are in running, and we will report the results as soon as possible.

• Q5: It's better to include the comparison with other data augmentation methods like MTDiff.

• A5: Thanks for your advice on comparing DyDiff with MTDiff. We use the official implementation of MTDiff and replace its task list by a single task. We first train the MTDiff-s model on the given dataset and then generate 1M $(s,a,r,s')$ samples as the synthetic dataset $\mathcal{D} _ {\mathrm{MTDiff}}$ . Finally, we train the RL policy on the mixture of $\mathcal{D} _ \mathrm{real}$ and $\mathcal{D}_\mathrm{MTDiff}$. The results are reported in Table 6, Appendix E.4. We find that MTDiff can hardly improve the performance of the underlying policy since it is originally designed for multi-task situations. Its generalization ability from learning knowledge across multiple datasets is significantly weakened in the single-task setting.

Thank you very much for addressing the clarification questions raised. I will maintain the score of weak acceptance.