What Makes a Good Diffusion Planner for Decision Making?

W1. The number of tasks used to investigate the design choices is limited.

W2. The paper does not verify the insights/findings on a different set of tasks (i.e., validation tasks) from those used in the design choice investigation ... leaves uncertainty about how generalizable the insights are.

Thanks for the suggestion. We have conducted experiments on additional datasets (Adroit, see Sect. 4.7 and Appendix C). The insights and conclusions drawn from the new results are actually consistent with the existing ones (i.e., the takeaways in Sect. 4.8). We have also included the code in the supplementary material to ensure reproducibility.

In particular, we observed that the following key findings are consistent. For Adroit tasks, the empirical results (Sect. 4.7 and Appendix C) show:

• It is better to generate a state sequence and then compute the action using an inverse dynamics model than to jointly generate state and action sequences, consistent with Sect. 4.1 (action generation).
• Jump-step planning is slightly better than dense-step planning, consistent with Sect. 4.2 (planning strategy).
• Transformer outperforms UNet as the backbone for the denoising network, consistent with Sect. 4.3 (denoising network backbone).
• A one-layer Transformer performs poorly, while there is no improvement with more than two layers, consistent with Sect. 4.4 (impact of network size).
• MCSS outperforms CFG and CG, consistent with Sect. 4.5 (guidance sampling algorithms).

W3. I didn’t quite understand the breakdown of these 6,000 diffusion models. Were most of these models the ones trained and evaluated through the grid search?

W4. What exactly does "manual tuning" refer to in this context?

We apologize for the lack of clarity regarding our hyperparameter search process. Training diffusion models demands significantly more computational resources compared to traditional Gaussian policies, making exhaustive grid searches impractical. Instead, we conducted several rounds of hyperparameter tuning, where each round focused on a subset of hyperparameters that we identified as most influential based on prior experiments and domain knowledge. The “manual tuning” refers to this iterative process of selecting which hyperparameters to explore in each round, guided by preliminary results and insights.

The "6,000+ models" is the cumulative number of experiments executed on our computing cluster throughout this iterative tuning process. These experiments include various combinations of hyperparameters tested across different rounds of searches. We have now included explanations of our hyperparameter tuning strategy in the revised manuscript (Appendix B.4) to provide better transparency.

W5. It seems that the Transformer score for Kitchen-M doesn't have a confidence interval.

W6. I wasn't clear on what the confidence intervals in the other parts of this figure represent (are they calculated based on 500 episode seeds?).

We appreciate your careful observation. In fact, the confidence interval for Kitchen-M exists, but it is very small and not visible on the plot. We have added an explanation in the caption of Fig. 5 to clarify this issue. Additionally, we have provided the numerical results in Table 5 in the Appendix.

Yes. These confidence intervals are calculated based on 500 episode seeds, representing standard errors.

Typoes:
line 101: Zhang et al., 2022) In -> Zhang et al., 2022). In
line 158: Chen et al., 2024)) -> Chen et al., 2024).
line 479: planning(Sect. 4.6) -> planning (Sect. 4.6).

Thank you for your careful reading. We have fixed these typos.

Finally, we would like to thank you again for the useful comments, which we believe have significantly enhanced our work. If you have any further comments, we would be glad to have a deeper discussion with you.

W1. Limited exploration of long-term dependencies: While the paper discusses the importance of handling long-term dependencies using Transformer, it does not delve deeply into how this is manifested across different tasks. The related discussion could be more robust.

Thank you for helping us strengthen our experiments. We have provided the attention maps for various tasks in Appendix D.1. Although the attention patterns vary across different tasks, they all exhibit long-term attention, suggesting that long-term dependencies are common among these tasks and explaining why Transformers outperform UNet. The attention patterns typically feature slashes, which attend to a fixed number of steps prior, and vertical lines, which attend to key steps. We have complemented Sect. 4.3 with these additional analyses.

W2. Potential typo in Equation 2.1: There seems to be a typo on the right-hand side of Equation 2.1, where S(t−1) appears, which might be incorrect.

Thank you for pointing out this typo. It should indeed be S(t+1), and we have fixed it.

Q1. You mention that unconditional sampling outperforms guided sampling, which contrasts with results in typical image generation tasks. Could you elaborate on the underlying reasons behind this phenomenon?

This is indeed an interesting question that warrants deeper investigation. In addition to the analysis provided in the manuscript (Sect. 4.5), we elaborate on the potential reasons from the following perspective:

Theorem 1: Suppose the reward distribution of the trajectories generated by unconditional models is identical to the reward distribution of the trajectories in the training dataset. Assume that, in the training dataset, the proportion of trajectories with a reward above $R$ is $p$. If we want MCSS's reward to exceed $R$, the expected number of sampling times is $\frac{1}{p}$.

Proof: We apply the total expectation formula to compute the expected value, denoted as $E$. For the first generated trajectory, if the reward is above $R$, the expected value is $1$. Otherwise, the expected value is $1 + E$. We then have $p + (1-p) \cdot (1 + E) = E$, which simplifies to $E = \frac{1}{p}$.

Using Theorem 1, we evaluated when MCSS can outperform guidance. We set the threshold $R = 0.9R_{\max}$. The expected number of sampling times for kitchen, antmaze, and maze2d are $66.66$, $6.45$, and $13.51$ (where $p$ can be obtained from the data in Fig. 7b), respectively. In the experiments, we set the number of sampling times to $50$. As a result, MCSS achieves better performance than diffusion guidance in antmaze and maze2d, but its performance is slightly inferior in the kitchen scenario.

Q2. The paper primarily focuses on state-based tasks. Are there plans to extend the study to vision-based or goal-conditioned reinforcement learning tasks?

This is a great point! Yes, we plan to conduct future studies on vision-based and goal-conditioned reinforcement learning (RL) tasks for real-world applications. We believe that combining proprioception (joint states), vision, and possibly force feedback/tactile sensing, along with goal-directed planning, will result in more powerful robotic AI.

Thank you again for your constructive feedback! We hope the responses above have addressed all your comments. Please kindly let us know if you have additional suggestions, and we would be more than happy to discuss them.

Thank you very much for reviewing our work and providing your thoughts.

While the paper provides strong evidence for the effectiveness of the proposed methods on the D4RL dataset, it is unclear how generalizable these findings are to other types of decision-making problems or datasets. More diverse datasets could strengthen the claims.

We appreciate your suggestion and agree that our work can be strengthened by incorporating more datasets. To address this, we conducted experiments using our model on additional datasets (Adroit, including 8 subtasks) to validate our findings, which have been discussed in Sect. 4.7 of the revised manuscript (detailed in Appendix C). For the experiments on Adroit, we inherited the hyperparameters used for Kitchen. The source code is included in the supplementary material and will also be published.

We observed that the following key findings are consistent. For Adroit tasks, the empirical results (Sect. 4.7 and Appendix C) show:

• Generating a state sequence and then computing the action using an inverse dynamics model is better than jointly generating state and action sequences, consistent with Sect. 4.1 (action generation).
• Jump-step planning is slightly better than dense-step planning, consistent with Sect. 4.2 (planning strategy).
• Transformer outperforms UNet as the backbone for the denoising network, consistent with Sect. 4.3 (denoising network backbone).
• A one-layer Transformer performs poorly, while there is no improvement with more than two layers, consistent with Sect. 4.4 (impact of network size).
• MCSS outperforms CFG and CG, consistent with Sect. 4.5 (guidance sampling algorithms).

We believe the above addresses your concern. Thank you for helping to improve our work. Should you have any other comments, please kindly let us know.

Thank you for reviewing our paper and raising the concern regarding "original innovation in theory." On one hand, we would like to emphasize that the controlled experiments presented in this paper have the potential to inspire future theoretical innovations in the field. For instance, one could develop a theory to provide insights into the surprisingly good performance of MCSS, as discussed in Section 4.5.

Theorem 1: Suppose the reward distribution of the trajectories generated by unconditional models is identical to the reward distribution of the trajectories in the training dataset. Assume that, in the training dataset, the proportion of trajectories with a reward above $R$ is $p$. If we want MCSS's reward to exceed $R$, the expected number of sampling times is $\frac{1}{p}$.

Proof: We apply the total expectation formula to compute the expected value, denoted as $E$. For the first generated trajectory, if the reward is above $R$, the expected value is $1$. Otherwise, the expected value is $1 + E$. We then have $p + (1-p) \cdot (1 + E) = E$, which simplifies to $E = \frac{1}{p}$.

Using Theorem 1, we evaluate when MCSS can outperform guidance. We set the threshold $R = 0.9R_{\max}$. The expected number of sampling times for kitchen, antmaze, and maze2d are $66.66$, $6.45$, and $13.51$ (where $p$ can be obtained from the data in Fig. 7b), respectively. In the experiments, we set the number of sampling times to $50$. As a result, MCSS achieves better performance than diffusion guidance in antmaze and maze2d, but its performance is slightly inferior in the kitchen scenario.

A similar theory could be developed to provide insights into the experiments in other subsections, which could lead to further performance enhancements in this field. However, complete theoretical justifications for the experiments are beyond the scope of this paper and are left for future work. Nonetheless, we believe our empirical results pave the way for future theoretical innovations in the field.

We hope the above addresses your concerns. Moving forward, we will carefully consider how to draw insights from theory to further strengthen our understanding of diffusion planning and decision-making.