Simple Hierarchical Planning with Diffusion

R1: Baseline Decision Diffuser (DD) is missing

Thank you for pointing out the Decision Diffuser (DD) omission as a baseline in our comparative analysis. We present the results in Table 1 below and will include DD in Table 2 in our revised paper.

Table 1. Comparison between DD and HD

R2: A comparison between using CNN and transformer architecture, such as Decision Diffuser, would be encouraged.

We appreciate the reviewer's constructive suggestion. As suggested, we compared our model (based on U-Net (CNN)) with Transformer, and the results are in Table 2 below.

Table 2: Comparison on HD-UNet and HD-Transformer

For this experiment, we use the hyperparameter setting in the Decision Transformer (DT) as a starting point for our investigation. The results, as shown in the table, reveal that the HD-Transformer achieves similar performance to the HD-UNet in Maze2D tasks, though it is slightly less effective in the Gym-MuJoCo tasks. While the HD-Transformer shows promise, we would like to emphasize that our primary contribution is not the backbone architecture but the benefits of hierarchical structures. Nonetheless, we are grateful for the reviewer's insightful and constructive recommendation. This ablation study will make the paper better.

Regarding the backbone architecture, we also did another related experiment suggested by another reviewer. Here, the goal is to see the performance of a deeper multi-scale U-Net as it can capture the hierarchical structure better within one architecture. As shown in Table 3 below, we see that HD still works significantly better. For more details, please see response R3 to reviewer zGcK.

Table 3: Comparison on Multiscale U-Net and HD

Q1: According to Fig 3., Diffuser with a small kernel size covers one mode of the data distribution during actual execution. Is it the case that the standard Diffuser is able to cover the full data distribution but just fails to achieve full data coverage during actual execution? If so, what might be the cause the such gap?

We appreciate the reviewer's query, which we also find interesting. The answer is no. We found that the diffuser generates plans in the bottom half and actual execution following this plan.

We think the chance that the actions deviate from the generated plan is low. This is primarily because the Diffuser model employs a PD controller to derive actions from the planned states, which include position and velocity. Consequently, the actual rollout is expected to follow the generated plan closely.

Q2: In Fig 3, HD fails to cover two short segments/branches. Why does such phenomena occur?

Thank you for highlighting this observation. It is a very interesting phenomenon to us as well. Our current best hypothesis is that it has something to do with ignoring actions in the Diffusion modeling. First, in another related work, AdaptDiffuser [1], which also only models states, we observe a similar smoothing effect in Figure 3. (a) and (b). Second, we observed that when we add actions in the diffuser (HD-DA), the smoothing effect is mitigated, as shown in the figure here. These suggest that the smoothing is highly related to ignoring actions in diffusion modeling. Thank you for pointing out this interesting question. We will discuss this in the revised paper.

R4: Goal-selection method is not flexible (only fixed time interval), may consider other methods such as spatial intervals and reward scale as in HDMI

Thank you for the reviewer’s suggestion regarding the flexibility of our goal-selection method. We ran additional experiments on several alternatives of sub-goal selection.

• Route Sampling[1] (RS): We also consider choosing waypoints with fixed length intervals as sub-goals in line with HDMI. Specifically, denote the distance moved after action $a_t$ as $\delta_t$. Then, the route length can be computed as $S = \sum_{t=0}^{T −1} \delta_t$ . We pick the waypoints with a fixed interval of $S/k$, where $k$ is the number of sub-goals.
• Value Sampling[2] (VS): Also inspired by HDMI, we test the value sampling method, where the most valuable states are chosen as sub-goals. Specifically, the distance-weighted accumulated reward is used to value each state after state $s_i$: $W(s_j) = \sum_{k=i+1}^j \frac{r_k}{j-i}$.
• Future Sampling[3] (FS): Beyond RS and VS, we also explored a hindsight heuristic method, randomly selecting future states as sub-goals.

Notably, in RS and VS, certain states might never be chosen as sub-goals, unlike FS and the fixed time interval sampling (TS) used in HD, which offers equal probability for each state to be selected as a sub-goal.

Given the varying lengths of sub-tasks generated by these selection methods, integrating dense action at the high level was impractical. Hence, we focused our experiments on HD rather than HD-DA. At the low level, sub-trajectories were padded to a consistent length $L$. It's important to note that excluding dense action data at the high level may hinder the learning of the value function, potentially leading to a marginal decrease in performance, as shown in Table 4 in the paper. As presented in the table below, the results demonstrate that our hierarchical framework is generally resilient across different sub-goal selection methods. While HD-VS and HD-RS exhibited somewhat lower performance, we hypothesize this may be due to uneven sampling of valuable states, which could impact the planning guidance function's effectiveness.

[1]: Lai, Y., Wang, W., Yang, Y., Zhu, J., and Kuang, M. Hindsight planner.

[2]: Correia, A. and Alexandre, L. A. Hierarchical decision transformer.

[3]: Marcin Andrychowicz, et al. Hindsight Experience Replay

Q1: Why only return-guided sampling is used in this paper instead of inpainting?

As shown in Appendix B, we utilized both value-guided sampling and inpainting, aligning with the approach used in Diffuser. Specifically, for Maze tasks, we employed inpainting as our sampling strategy. In the case of more complex controlling tasks, we augmented this with value guidance.

R1: Highlight the algorithm’s generalization performance more

We thank the insightful feedback. Following the suggestion, we conducted an additional experiment for the OOD generalization task. Please see response R4 for reviewer zGcK for the details.

R2 Considering HDMI, contribution is limited in this work.

Thank you for your feedback regarding the perceived limited contribution of our work in comparison to HDMI. While there are certain similarities between our HD framework and HDMI, the differences in focus, methodology, and resultant performance and efficiency gains highlight the substantial contributions of our work. Overall, our paper leads to a different lesson from HDMI because our paper suggests that the high complexity of HDMI is actually not required; we can still achieve even better performance with a much simpler approach.

In the following ,we respectfully offer a different perspective, highlighting several distinct aspects of our work:

• Different methodologies:
  • Problem formulation: HDMI casts the offline decision-making as a conditional generative problems where sampling is conditioned on the pre-defined desired return. Our method formulate the decision-making problems as sequence modeling problem. Policy is improved via the classifier guidance sampling, which guides the planner to generate plans that maximize the expected return.
  • Sub-goal selection: HDMI’s sub-goal preprocessing requires explicitly building a graph on the training dataset and applying costly graph search. Our method works robustly with simple fixed time interval heuristic method, which is more computationally efficient. HDMI’s subgoal selection is very complex. This process requires components like (1) training a separate RL policy like IQN or D4PG only for subgoal selection, (2) building a weighted directed graph, (3) dataset reduction by running K-means++ algorithm, (4) learning goal-conditioned value function, (5) distributional RL, (6) running Floyd-Warshall algorithm to find shortest path, (7) low-temperature sampling to prevent “same sub-gaol issue”, and so on. Also, code is not available. These are all used only for goal selection.
  • Low-level planning: HDMI low-level planner is ignorant of the task rewards, which might lead to local sub-optimal. Our low-level planner is guided with both sub-goals and the task rewards.
  • Policy inference: HDMI models only the states sequence, actions are inferred from an inverse dynamic module. Our method models both states and actions directly.
  • OOD generalization: We presented one compositional generalization task in our paper, the solvability of which by HDMI remains unclear. During rebuttal, we provide an additional OOD collision avoidance task, a challenge that conditional-generative methods like HDMI can not address effectively, even with ground-truth reward function. This is because the OOD obstacles introduced novel reward structures that will shift the distribution of trajectory returns, i.e. a novel trajectory return value has never been seen during training, making return-conditioned generating fail. However, the classifier guidance used in our method can guide the plans gradually move away from collision.
  • Theoretic analysis: We also provide theoretical analysis which fits well to the proposed model due to the simplicity of the proposed method.
• Superior Performance: Our empirical results, as shown in Table 1 and Table 2, HD significantly out-performs HDMI on both the long-horizon goal-conditioned tasks and Gym-MuJoCo tasks.

R3: Demonstration of the quality of the generated goals is missing, which is essential.

We thank the reviewer for your valuable feedback highlighting the need to demonstrate the quality of the generated sub-goals. To address this, we have included visualizations of randomly sampled plans from the Maze2D tasks, which can be found here. In these visualizations, the sub-goals are highlighted with large circles. These illustrations clearly demonstrate how the sub-goals effectively guide the low-level plan towards the overall goal. We appreciate the opportunity to enhance the clarity and depth of our presentation.

R1: Limited novelty

• Thank you for your comment regarding the perceived novelty of our formulation. While it is true that the core concept of using diffusion models for planning is not entirely new, the novelty of our work lies in the specific implementation and integration of these models in a hierarchical planning framework. The integration of a sparse diffuser with a standard diffuser is a novel aspect of our work. Also, our proposed method is very simple and thus practical compared to other method like HDMI. However, we show that this simple model achieves even better performance than the previous complex model.

R2: Why states-only modeling (as in Decision Diffuser) are not sufficient for having a good estimation of $J$?

• In our model, state-only modeling is not enough because we also skip intermediate states in our high-level planner. We think it is not a problem in Decision Diffuser because they do not skip intermediate states. Without knowing what has happened in the skipped intermediate states, the value estimation can be difficult.

R3: Is sampled action or controller used in Maze2d tasks like Diffuser?

• The same controller is used as in Diffuser on the Maze2D tasks for a fair comparison.

R4: Solving sub-goals independently might lead to non-optimal behavior

In the replanning mode of our method (the closed-loop mode), the low-level planner does not independently solve the segment because the high-level planner also updates the subgoal depending on the state of the low-level planner. In the open-loop mode, which we use when the task can be segmented more independently, we rely on the quality of the high-level planner to connect the low-level plans.

R5: How is overlapping avoided during low-level sampling?

The planing at the low-level is a goal-conditioned sub-task. The low-level planner takes in every two consecutive sub-goals in the high-level plan as starting state and goal state, and is trained to fill in intermediate states. Thus the low-level plans can be assembled according to the sub-goal order without overlapping.

R6: How many trajectories are sampled for individual segments and sub-goal sequences?

For goal-conditioned long-horizon tasks, one trajectory is sampled for both individual segments and sub-goal. For the Gym and Kitchen tasks, 64 trajectories are sampled.

R7: What happens when sub-optimal action diverges from the sub-goal for the maze case?

When sub-optimal action diverges from the sub-goal, resulting in an unexpected state, we correct the bad effect of sub-optimal actions by replanning. Increasing the replanning frequency improves the performance because the plan gets the chance to be corrected. It’s a trade-off between performance and computation resources. Other alternatives might also be worth exploring, i.e., explicitly learning a goal-conditioned policy $\pi_\theta(a|s, s_g)$ might be beneficial when the state and action space are high-dimensional, bringing challenges to modeling the whole sequence.

R8: Contributions are not significant

Thank you for your feedback regarding the perceived limited contribution of our work. We respectfully offer a different perspective, highlighting several contributions which has also been acknowledged by other reviewers:

• Improvements in Performance and Efficiency with a Much Simpler Model: Our work with the HD framework has demonstrated notable improvements over our previous baselines in performance and computational efficiency. These advancements are substantial, as they contribute to our approach's effectiveness and practical applicability in real-world scenarios. This improvement has been recognized and underscored by reviewer AgLN. Our method is also much more simpler than the previous hierarchical diffusion method, HDMI.
• Theoretical Analysis and Experimental Validation: We have proposed a novel approach and supported it with rigorous theoretical analysis and well-designed experiments. This comprehensive approach ensures that our contributions are grounded in theory and practice, providing a robust foundation for our findings. (reviewer AgLN)
• Insights into Receptive Field and Data Coverage: Another significant aspect of our work, as highlighted by reviewer KFqH, is the identification of the relationship between the diffusion model’s receptive field and data coverage. This insight is particularly valuable as it opens new avenues for further research in the field.

In summary, our contributions extend beyond mere incremental improvements. The combination of performance enhancements, theoretical analysis, and new insights into key aspects of diffusion models collectively represent significant advancements in the field. We are confident that our work offers valuable contributions and lays the groundwork for future research and development.

Dear Reviewer zGcK,

It seems that you missed providing the title of the papers [1] and [2]. Thank you!

Sorry, my bad.

[1] AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners

[2] Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models

R1: Current SOTA has a better performance, e.g. [1] 167 score on large mazed2d

We respectfully disagree with the statement that AdaptDiffuser is considered the current state-of-the-art (SOTA) method superior to ours. Comparing our method directly with AdaptDiffuser is unfair due to several key factors.

• First and foremost, it is unfair to compare our method with AdaptDiffuser [1] because AdaptDiffuser relies on using privileged knowledge, specifically the true dynamics function $\mathcal{T}(s, a)$. This strong assumption is usually not made in this line of research, which our method and the baselines are focusing on.
• Secondly, it would not be appropriate to claim that AdaptDiffuser is superior to our method based only on the performance of a specific task. Instead, one should consider the overall performance across multiple tasks rather than a single task. In the DMC MuJoCo benchmark, our method outperforms the AdaptDiffuser (despite it using the true transition dynamics) in 5 out of the 9 tasks. Ours has a higher average score (84.6 for ours vs. 83.4 for AdaptDiffuser). Even in cases where AdaptDiffuser outperforms our method, the difference is small (e.g., 38.1 for ours vs. 38.3 for AdaptDiffuser in Med-Reply HalfCheetah). Additionally, it is worth noting that the AdaptDiffuser paper does not evaluate the Multi2D tasks, which are considered a standard benchmark for diffuser-based planning models. Therefore, it is unfair to conclude that AdaptDiffuser is generally the SOTA and better than ours based on the performance on large maze2D.
• Lastly, we would like to emphasize that the contributions of the two papers (ours and AdaptDiffuser) are orthogonal. AdaptDiffuser introduces a data-augmentation approach, while our work introduces a hierarchical approach. These two ideas are not in competition but can be used in conjunction. We believe both ideas are valuable to the community, and there is no need to choose one. Exploring the synergy of these two ideas is an interesting avenue for future work.

We hope this clarifies our position and highlights the unique contributions of our work. We appreciate the reviewer's feedback and the opportunity to address their concerns.

[1] Zhixuan Liang, et al. AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners.

R2: Improvement in MuJoCo is not enough to me

We deeply appreciate the valuable feedback provided by the reviewer. However, we respectfully disagree with the subjective judgment that there was "not enough" improvement in the MuJoCo benchmark. We would also like to kindly request further clarification on what "not enough" means more technically and scientifically.

Nevertheless, we firmly believe that the improvement achieved in the MuJoCo benchmark is both meaningful and significant. Our method consistently outperforms all baselines, with an average score of 84.6 compared to 77.5 for Diffuser and 82.6 for HDMI. Although the improvement over HDMI may not appear substantial at first glance, it is crucial to note that this averaged score is calculated across 9 tasks, highlighting the consistent performance enhancement of our approach. Furthermore, we are proud to emphasize that our method achieves these results while maintaining a significantly simpler and faster framework, particularly compared to the HDMI baseline.

Once again, we sincerely appreciate the reviewer's valuable feedback.

R3: The Unet itself has hierarchical structure. Why not simply increase the depth of Unet? Provide more result on this.

Thank you for the insightful feedback. We agree that this is an interesting observation and worth investigating. Following the suggestion, we conducted an additional experiment comparing a Diffuser with a deeper Multiscale U-Net (with twice as many layers) and HD. As shown in Table 1 below, we can see that HD significantly outperforms the deeper Multiscale U-Net. Furthermore, we found that the deeper Multiscale U-Net is computationally much more expensive, leading to very slow training and test times (Table 2 and Table 3 below). More details about the experiment can be found in the Appendix of the updated paper, which will be uploaded.

Table 1: RL Performance Comparison on Maze2D

Table 2: Training Wall-Clock Time (sec.) Comparison on Maze2D

Table 3: Testing Wall-Clock Time (sec.) Comparison on Maze2D

R4: The OOD task is too simple. Harder OOD tasks are tested in other papers. [1] add coins [2] add obstacle

Thank you for the feedback. Based on your suggestion, we conducted an additional out-of-distribution (OOD) experiment on one of the suggested tasks, specifically the obstacle task used in the MPD model [1]. Following the methodology of MPD [1], we applied our HD model to the PointMass2D Dense task with OOD obstacles. To do this, we replaced the flat Diffuser planner used in MPD with our HD model, referred to as MP-HD. We used the official code of MPD for comparison.

As shown in Table 4 below, our HD model performs significantly better than MPD on these challenging OOD tasks. We also visualized sample trajectories of HD from two randomly selected (start, goal) pairs here. These trajectories demonstrate that our model can effectively avoid collisions when faced with OOD test obstacles.

(Note that the official MPD code yielded a success rate of 75.0 ± 43.3, which is lower than the performance reported in the MPD paper (79.0 ± 40.7). In a personal conversation with the author, we were informed that reproducing the score in the paper using the official code is currently challenging for some reason.)

Table 4: RL Performance Comparison on PointMass2D Dense - Extra Obstacles

[1]: Joao Carvalho, et al. Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models

R5: It is important to ensure smoothness on the connection points. The paper didn’t talk about it.

Thank you for the constructive feedback. While we understand the significance of this aspect, we would like to clarify that our method has been empirically observed to exhibit robustness in terms of smoothness at the connection points because, in our experiments, we have achieved high performance without introducing any specific means to address this issue.

In our approach, the inpaint (open-loop) mode ensures smoothness as we perform inpainting to fill the states between the subgoals suggested by the high-level planner. This creates a natural flow, where a subgoal becomes the starting state for the next subgoal, resulting in smoothness at the connection points. Additionally, in the replanning (closed-loop) mode, the high-level planner continuously replans the subgoals at each time step, eliminating the need to be concerned about smoothness at the connection points.

We appreciate the reviewer's feedback on this topic, and based on their input, we will further discuss and elaborate on this aspect in the revised version of our paper.

Q1: I don’t understand the meaning of SD-DA. It seems like only improves the performance of walker2d. I feel like predicting so detailed actions loses the benefit of high-level planning.

SD-DA is a model for ablation study to help understand what aspects of our final model contribute to the performance. The meaning of this model is written in detail in the paper as follows.

The missing states and actions in the subsampled trajectories $x^\text{SD}$ might pose difficulties in accurately predicting returns in certain cases. Therefore, we investigate a potential model improvement that subsamples trajectories with sparse states and dense actions. The hypothesis is that the dense actions can implicitly provide information about what has occurred in the intermediate states, thereby facilitating return prediction.

Table 4 in the paper shows that including dense action significantly improves value estimation performance. Additionally, it contributes to RL performance, although the degree of improvement varies across different tasks. Notably, walker2d shows the most significant improvement, while the effect on other tasks is less pronounced.

In conclusion, these experiments serve as an ablation study to gain a better understanding of the model. We believe that such analysis provides crucial knowledge to the community.

R1: Thanks for clarifying the two papers. To me, the transition function is not hard to get (or train) in a mazed2d. I believe a better way to show your method is to combine this two together and get a even better SOTA.

R2: 84.6 and 82.6 is really just not too impressive.

R3: Thanks for the experiment. It address some of my concerns.

R4: Thanks for the experiment. It address my concerns. It will be great if you also have the result for the coins.

Q1: Since it can only improve one of third environments, it is not that clear if it is something that can improve the method consistently.

I have raised the rating from 3 to 5 due to R1, R3, and R4.