We appreciate the thorough review and the valuable suggestions you've provided regarding the clarity of the state encoder, dataset optimality, and the usage of T5-XXL, each of which has helped us to reflect on and improve the manuscript. We would like to address each of these points specifically.

Weaknesses:

1. It appears that some crucial descriptions of the model are missing. For instance, there is little mention in the paper about how the state encoder is trained. After going through HULC[Mees et al., 2022a], I just noticed that the encoder is possibly the one in HULC model. Therefore, the author considers both the encoder and LLP to be pretrained. I suggest the author add the corresponding description because it is currently hard to find out this from the logical flow of the paper. It is still important to clarify the training objective and the structure of the state encoder and low-level policy.

We have added a new section in Appendix C in the paper to help clarify the choice of state encoder.

We confirm that the state encoder is not explicitly trained, but rather is just the output of a hidden layer within the low-level policy. We attempted to illustrate this in Figure 1, although recognize this may have been unclear.

We have tried our best to clarify our definition and choice of state encoder, and now reference Appendix C in the introduction, the end of the background, and in Section 3.2.

2. The novelty of the paper is limited, as it is a combination of diffusion model and HULC.

We respectfully disagree with the reviewer's assessment. It is nontrivial to combine diffusion models as a high level policy (HLP) in hierarchical RL with a low level policy (LLP).

Naively combining the two immediately leads to several problems - choosing a principled representation for the diffusion model to output becomes an issue, and simply training end-to-end would lead to a nonstationary objective for the diffusion model (as the data distribution is changing) [1]. We have this ablation in Table 1 as Diffuser-1D and Diffuser-2D, which significantly underperform all other models. On the other hand, trying to avoid these issues by training the diffusion model directly to output videos is computationally expensive [2] and also models task-irrelevant details, which may lead to failure in generalization [3] (Please see their Appendix A.1. It is not obvious how to simultaneously solve the nonstationarity, efficiency, and representation problem.

We propose a novel solution to the representation and efficiency problem by proposing the usage of a frozen pretrained LLP representation. This is not an obvious solution. Furthermore, we give a principled motivation for this choice with our suboptimality theory in Proposition 3.1. Finally, we solve the nonstationarity problem by proposing a latent two-stage training approach, by freezing the LLP during the HLP diffusion training.

To the best of our knowledge, no other paper has proposed or instantiated a successful latent diffusion planner, let alone with language conditioning. In our work, we are able to demonstrate both the theoretical and empirical effectiveness of such an approach. Empirically our model significantly outperforms all other methods, and does so 3.3x to 15x faster than other diffusion-based approaches.

[1] Sohl-Dickstein et al. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML, 2015. [2] Rombach et al. High resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022. [3] Du et al. Video Language Planning. arxiv, 2023.

3. The assumption of the dataset being optimal is very strong in real-world settings, which limits the applicability of the proposed method.

Empirically we have found that our method works well even when this optimality assumption is violated, as the data in the CALVIN benchmark is in fact suboptimal [4]. We do acknowledge that there is a gap between the theory and practice and have now explicitly detailed this gap in the paper here in Appendix G.

[4] Mees et al. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 2022b.

4. The textual encoder - T5-XXXL model in this paper - is quite large, which increases the inference time. Is it possible to use the textual encoder in CLIP?

Thank you for the suggested ablation. We are currently running these experiments now, and will update you when they have finished running.

4. Besides, I would like to know if these text-to-video methods are directly comparable to LCD in experiments.

We give two main points of comparison between text-to-video methods and LCD here:

1. Robustness of Communication Interface: Text-to-video methods rely heavily on a high-quality decoder or skill oracle capable of generalizing well across various contexts (see Appendix A.1 of Video Language Planning). The quality of these components is critical for their success, and performance can degrade if these elements do not generalize well. Our approach incorporates this inductive bias and performs diffusion directly in the latent goal space of the decoder rather than in the image space, which often hallucinates (see Figure XII of Video Language Planning, related work in Tune-A-Video [5]) and is significantly more computationally expensive. Thus, our method potentially offers more consistent results in a wider range of scenarios.

2. Efficiency and Accessibility: Our LCD method was designed with efficiency in mind. Text-to-video methods may indeed give better performance in some scenarios where large-scale compute is accessible. However, this is often not an option for individual researchers or small-scale labs. LCD provides an agile and less resource-demanding alternative, enabling rapid iteration and experimentation with diffusion planners.

In conclusion, when comparing with text-to-video methods, LCD offers a valuable and pragmatic solution for research settings where computational resources or the ability to develop complex decoders and communication interfaces are limited. We believe that the LCD's advantages, particularly in terms of efficiency and robustness, make it a worthwhile alternative for many practical applications.

[5] Wu, Jay Zhangjie, et al. "Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

5. It is not very clear to me what dom(P(s'|s, a)) measures and the intuitive sense behind it. For terms that are not commonly used in RL theory, it would be better to state its definition and intuition of the proof/bound. It is also not clear what the circle means in the definition of the low-level policy.

$\text{dom}(P(s'|s, a))$ is the size of the set of possible next states, or more formally the cardinality of the domain of the transition dynamics function. In the definition $\pi_{lo}(s) := \phi \circ E(s)$ , the $\circ$ symbol refers to function composition. We have added these definitions for clarity in Section 3.1.

7. The theorem comes before the practical instantiation of the algorithm. The gaps between theory and practice should be explicitly analyzed.

We appreciate the reviewer's thoughtful recommendation. Following this guidance, we have elaborated on this aspect and included a new section in Appendix G of our paper. We trust this addition clarifies the point and strengthens our argument.

We believe that our comprehensive rebuttal and modifications emphasize the strength and scalability of our research, with the capacity to influence future advancements in the field. We hope that our responses have sufficiently met your queries and concerns. This paper's potential to enhance scalability and advancement toward longer horizons could greatly contribute to this research area. Thank you very much for your thoughtful evaluation of our research, and we eagerly look forward to your continued feedback.

We appreciate the reviewer's insightful questions and valuable critique. To summarize, we have addressed concerns about the role of low-level controllers, differences between our method and other text-to-video works, the comparability of these methods in experiments, the clarification of certain terms and symbols, and the gaps between theoretical and practical applications in our approach. We trust that our responses below will provide a more comprehensive understanding of our research and its implications.

Weaknesses:

1. The authors stated that the proposed algorithm avoids "the usage of a predefined low level skill oracle". Isn't the low-level controller also adopted by LCD in this work? Do the authors refer to the imitation learning setting where we only access to the trajectories instead of the controller?

Here, we refer to two disparate assumptions that we avoid: > 1. We do not have a perfect low-level controller or inverse dynamics model. This is exactly as you say above. Instead, we must learn a low-level policy (LLP) in our setting. > 2. We do not define a fixed set of predefined skills between the high-level policy and the low-level controller, also known as the "communication interface". For example, in Saycan [1] the high level policy chooses between a set of skills like {find apple, place coke, go to table}. In our setting, we do not assume that such a set has been defined.

[1] Ahn et al. Do as i can, not as i say: Grounding language in robotic affordances, 2022. https://arxiv.org/abs/2204.01691

3. Can the authors comment more on the difference between the proposed method and previous text-to-video works, such as [1, 2]? Again, the authors mentioned that "they again avoid the issue of directly generating low level actions by using a low-level controller or an inverse dynamics model which we are able to directly solve for", but I didn't see the the downside of leveraging low-level controllers.

[1] Learning Universal Policies via Text-Guided Video Generation.
[2] VIDEO LANGUAGE PLANNING.

Thank you for raising this important point. Here are two reasons why avoiding low-level controllers may be advantageous:

1. Non-Transferability Across Domains: Many domains do not have a readily available predefined low-level controller, and constructing such a controller requires extensive work that is domain-specific [3]. Once built, these controllers are often not transferable to other contexts, limiting their applications to the scenarios for which they were originally designed. Predefined controllers may restrict the ability of a model to generalize to new contexts or tasks. They are often tailored to specific skill sets and may not support unanticipated actions or sequences that were not considered during their development. This limitation becomes evident when we encounter a requirement for a novel skill or action that the existing low-level controller does not support.

2. Complexity in Construction: A low-level controller can be intricate to construct [4], particularly in environments that involve many degrees of freedom or require nuanced motor control. The development process can be labor-intensive and may require expert knowledge in robotics, motor control, or specific domain physics.

[3] Hatze et al. The fundamental problem of myoskeletal inverse dynamics and its implications

[4] Hitzler et al. "Learning and Adaptation of Inverse Dynamics Models: A Comparison," IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), Toronto, ON, Canada, 2019, pp. 491-498, doi: 10.1109/Humanoids43949.2019.9035048.

We defer our answer on the difference between our method and previous text-to-video works below.

fAWH

We thank the reviewer for their perceptive questions and feedback on the performance of inference using the MLP and Transformer ablation, considerations of parameter size in making comparisons, and the role of HULC as a baseline in LCD. In the responses following, we have sought to address each point in detail. We hope our clarifications will elucidate a better understanding of our approach and its potential to influence this research field.

Weaknesses:

1. Additionally, I did not notice a detailed explanation of how the authors perform inference using MLP and Transformer.

Regarding the inference procedure using MLP and Transformer, we have since added a new section in Appendix E that provides a detailed explanation. Appendix E also includes a more detailed treatment of the hyperparameters for the transformer ablation.

In Section 4.5 of the experiments, the parameter sizes of MLP and Transformer are significantly smaller than the parameters of Diffusion. This may introduce unfairness in the experiments.

We would like to note that for this experiment we rigorously tested several model sizes for the Transformer to ensure a fair comparison, ultimately reporting results for the most performant Transformer; this Transformer has a smaller parameter count than our Diffusion model. The hyperparameters for this gridsearch can be found in Appendix E.2. However, we found the performance of the Transformer is still significantly lower than LCD, as evidenced in Section 4.5. Notably, we have since run further experiments with a similar-sized MLP (25.5M params vs LCD 20.1M params) that further reinforce our original findings.

Since the authors employ a Diffusion planning paradigm as HLP, it may be worth considering a better comparison with planning-based MLP (e.g., model-based RL) and Transformer (e.g., Decision Transformer).

In this setting, we strove to give the ablations as much advantage as possible, and in order to keep simplicity and ease the learning objective for the ablations we did next-state prediction rather than planning. Generally, it has been found empirically that the MLP does not work well in MBRL and trajectory generation [1, 2], so we chose only next-state prediction as the objective. We gave the same objective for the transformer to ensure uniformity.

[1] Janner, et al. Offline reinforcement learning as one big sequence modeling problem, Advances in neural information processing systems 2021.

[2] Chen, et al. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 2021

Questions:

3. The experimental results seem to indicate that HULC plays an important role in the success of LCD. Do the authors believe that the performance of LCD necessarily relies on a well-trained HULC baseline? Can LCD still achieve excellent performance with a simple goal-conditioned policy and a well-designed RL encoder representation? Alternatively, if a HULC baseline is used, can a Transformer with a parameter size comparable to the Diffusion HLP achieve similar performance?

We would like to clarify that we used HULC for our low-level policy for two main reasons. First, we aimed to leverage the strongest existing baseline to maximize our chances of creating a SOTA model. Second, we encountered issues replicating the results for the GCBC flat policy from its original paper, making it less favorable for our approach. It is important to note that there is, in principle, no reason why LCD cannot work with a flat policy, and this would be a straightforward addition for future work. We kindly refer you to Appendix C, where we have also explicitly clarified this point in our paper.

We want to reiterate that we did not find a Transformer with a comparable parameter size that was able to achieve similar performance to LCD.

---

We are confident that our thorough response and amendments underscore the robustness and scalability of our work, potentially driving future progress in the field. We trust that these responses adequately address your concerns. We believe that this paper's potential to further scale to longer horizons and improved generalization could make a significant impact on this research topic, and are extremely grateful for your review of our work. We look forward to your continued feedback.

We are grateful to the reviewer for their valuable comments, which allows us to emphasize the novelty of our diffusion model framework. We also justify our choice of the CALVIN benchmark as a testing ground and respond to requests for more extensive experimental evidence.

Weaknesses:

The weakness comes from a combination of lack of originality and broadness of the experiments. While the approach is very reasonable (applying language-conditioned latent diffusion in the abstracted state and action space for high-level planning), its general idea is similar to the ones in the literature (e.g., [1]). The major differences are the use of diffusion models and the specific choice of temporal abstraction.

We respectfully disagree with the reviewer's assessment. We would like to clarify that our main novelty comes from being the first to propose utilizing the implicit encoder of a pretrained low level policy for use in a latent language diffusion model and demonstrating both the effectiveness and efficiency of such an approach. On the other hand, PARROT [1] does not incorporate language and therefore must learn a new high level policy per task. In addition, it uses a generative model (normalizing flow) for the low-level policy (LLP), which is a more restrictive class of models than general low level policies. We are able to generalize zero-shot to new tasks through language, and do not need to make such assumptions on the LLP.

Our method innovatively integrates diffusion models into hierarchical RL as a high-level policy, overcoming common pitfalls. A naive approach combining the two grapples with an unprincipled nonstationary objective [2] if trained end-to-end, which we show in Table 1 as Diffuser-1D and Diffuser-2D. These baselines significantly underperform all other models. Alternative strategies, like two-stage training a text-to-video diffusion model after training the LLP, are not only resource-intensive but also capture unnecessary details that impede generalization [3, 4].

Our approach crafts a novel solution to these challenges by leveraging a frozen, pretrained low-level policy representation – a non-obvious choice that we defend with our suboptimality theory (Proposition 3.1). We further address the nonstationarity issue with a latent two-stage training protocol, enhancing both efficiency and performance. Empirically our model significantly outperforms all other methods, and does so 3.3x to 15x faster than other diffusion-based approaches.

To the best of our knowledge, no other paper has proposed or instantiated a successful latent diffusion planner, let alone with language conditioning. In our work, we are able to demonstrate both the theoretical and empirical effectiveness of such an approach.

[1] Parrot: Data-Driven Behavioral Priors for Reinforcement Learning [2] Sohl-Dickstein et al. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML, 2015. [3] Rombach et al. High resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022. [4] Du et al. Video Language Planning. arxiv, 2023. (Please see their Appendix A.1)

Based on this, I would like to see more experiment evidence, e.g., beyond the CALVIN benchmark, or real-world experiments (as in SPIL).

Our work stands out as the first to leverage a pretrained latent encoding within a new diffusion planning framework, offering unique efficiency and effectiveness as substantiated by our results.

We would like to emphasize that we believe the current experimental evaluation is sufficient for showing the effectiveness of our LCD approach. The choice of the CALVIN benchmark was deliberate; it is a well-established and challenging benchmark that tests for both the efficacy and efficiency of high-level planning and control in a complex environment. Our results on this benchmark clearly demonstrate the practical benefits of the proposed LCD method over existing techniques, thereby validating our contributions despite the perceived overlap with earlier works.

Moreover, we are also aware of the imperative to demonstrate the generalizability and applicability of LCD. To this end, we are in the process of setting up additional experiments on the CLEVR Robot Env and will update you later with these findings. These will further substantiate our claims and illustrate the versatility and robustness of the LCD framework beyond the CALVIN benchmark.

In conclusion, while we acknowledge the reviewer’s call for further experimental evidence, we maintain that our current experiments are both relevant and convincing for the scope of this study. Additional experiments, though underway, should be seen as extending rather than essential to the core contributions of the work.

Dear z6e9,

We wish to bring to your attention that we have posted new benchmark results, providing new data that may have a bearing on the evaluation of our work. These results have been rigorously obtained and we believe they contribute positively to the overall strength and validity of our research.

We trust that the supplementary results and thorough answers provided sufficiently address the issues highlighted, prompting a reevaluation of the initially given score. Thank you very much for your time and thoughtful consideration. We are grateful for the expertise and effort you bring to the review process.

CLIP Ablation Results

We have ablated the T5-XXL choice of encoder now by running LCD with CLIP embeddings, and include the results below. We use the same training methodology as in our original paper for Table 1, and rigorously follow the CALVIN benchmark’s evaluation procedure by running 3 seeds with our method, each seed evaluating for 1000 episodes. Numbers for HULC with CLIP are taken from the original author's paper [1].

---

Table 1: Comparison of HULC and LCD performance metrics

---

Across all horizons, our method (LCD) significantly outperforms HULC, as indicated by the heavier weighting in bold. In addition, we find that the CLIP embeddings are not as performant as the T5-XXL embeddings, indicating that CLIP potentially leads to worse generalization across language for robotic tasks than T5-XXL.

[1] Mees et al. What matters in language conditioned robotic imitation learning over unstructured data, 2022a. URL https://arxiv.org/abs/2204.06252.