We thank Reviewer JZP2 for their thorough review. We seek to address their listed considerations below:

• On soundness: the reviewer raises the question of how diffusion models can be applied to frames produced by game environments, if the diffusion models are trained on natural images. Indeed, the diffusion models are trained on natural images; however, they are also trained on cartoon images, paintings, movie posters, and more. We believe that the large-scale, pretrained diffusion model is able to encode the essence of the environment.

• On avoiding getting stuck: we provide analysis into why leveraging a text-to-image generative model as a reward signal does not cause the resulting policy to get stuck in achieving some best-matching frames in Appendix A.1. On the other hand, this is a common and natural pitfall when using alignment models such as CLIP as the supervisory reward signal to train a policy, as we both quantitatively and qualitatively demonstrate throughout the paper (in particular Walker, as well as the new Dog experiments). We believe that the stochasticity stemming from the source noise causes the pretrained, frozen diffusion model to constantly change its current belief of what the "best-matching frame" is. However, this does not result in a wildly unstable reward signal; as the diffusion model has learned a conditional distribution, these changing "best-matching frame" beliefs will always have high alignment with the provided text prompt. Ultimately, for continuous locomotion prompts, where there is a large set of aligned images (e.g. for "walking" there are many valid poses of different configurations of swinging arms and legs), we find that TADPol learns to perform coherent, text-aligned motion. On the other hand, as CLIP is deterministic, there is inherently a singular frame achievable by the policy that will have the highest alignment score with the text prompt. Then, a policy optimized using a pretrained CLIP checkpoint will learn to seek and get stuck maintain that best-matching frame.

• On using video models: we would like to mention that we do compare against video methods: we perform experimentation of our approach against Vid-TADPol, which uses a text-to-video diffusion model in place of a text-to-image one for TADPol. However, we surprisingly find that Vid-TADPol achieves comparable performance with TADPol in our experiments. One potential explanation might be that learning to walk in the OpenAI Gym environment is relatively insensitive to the priors encoded in the text-to-video model beyond those found in a text-to-image model. Another potential explanation is that Vid-TADPol is limited by the quality of currently existing text-to-video solutions. Addressing this finding and exploring how natural videos can be used to supervise the learning of novel text-conditioned policies is exciting future work.

• On the complexity of video demonstrations: we provide additional results demonstrating TADPol on the complex Dog environment from the DeepMind Control Suite. In these experiments, we show that TADPol is able to learn complex procedures (such as having the Dog stand on its hind legs or chase its tail). Furthermore, we showcase in our updated experiments how TADPol can learn motion policies.

• On comparisons to language-guided tasks: in this work we focus our efforts on learning text-conditioned continuous locomotion capabilities, and we show evidence suggesting TADPol is a promising approach towards this objective. Text-conditioned continuous locomotion is difficult compared to goal-achieving tasks (including language-guided navigation) in that there is no canonical pose or goal frame that, if reached, would denote successful achievement of the task. For example, there is no specific pose for “running” that when reached, would successfully represent completion of the “running” behavior; instead, agents must perform a coherent continuous sequence of poses over time (such as alternating stepping with the legs and swinging the arms) to convincingly demonstrate accomplishment of the task. To our knowledge, there are no prior works that seek to tackle this objective. Furthermore, the environments of many language-guided tasks are egocentric in nature; we leave it as interesting future work to explore whether or not pretrained diffusion models naturally understand how intermediate egocentric frames should be aligned with a desired text instruction.

Thanks for your response. I also understand the effort of the authors to publish this paper in time. Some of my concerns are addressed, and the response reads reasonable. So I'll upgrade my score by one.

We thank Reviewer JZP2 for the prompt and informative replies, particularly this late in the discussion period, and seek to address their concerns.

On the difficulty of the Dog environment: we provide two references, [1] and [2], that testify to the difficulty and complexity of the Dog environment, particularly due to its large 38-dimensional action space and its complex transition dynamics. Concurrent work (https://openreview.net/forum?id=Oxh5CstDJU) demonstrate that SAC and DreamerV3, powerful model-free and model-based approaches established in the field, are unable to learn any of the Dog tasks despite training on the ground-truth reward (Figure 5). We selected the Dog environment to demonstrate TADPol's capabilities because beyond being complex, it naturally enables us to flexibly demonstrate a variety of goal-conditioned and continuous locomotion behaviors. We agree that robotic environments are similarly worthwhile and interesting to apply TADPol to, and leave it as interesting future work.

On comparisons with RL: We agree with the reviewer that our method is RL that uses external supervision, and we do indeed compare against other externally-supervised RL techniques (the authors believe that "unsupervised RL" as a term might more accurately describe curiosity-based or intrinsic motivation-based RL, where the agent generates its own reward signals). We also agree with the reviewer that comparison with standard/ground-truth RL is valuable; and indeed we do already provide such numerical reports in Column 2 of Table 1 of the paper. For further reference, in the updated experiments on the Dog, the ground-truth TD-MPC model achieves about 150 reward, whereas our hybrid TADPol Dog agent achieves a comparable reward of 138; the rest of the ground-truth rewards achieved by externally-supervised approaches are listed in Table 3. Quantitatively, we show that TADPol is able to achieve high ground-truth reward despite using an external text-conditioned reward as a supervisory signal; qualitatively, we show that it can indeed recreate the behaviors described by the original ground truth reward function. We thus believe TADPol is comparable to ground-truth RL techniques. We also note that ground-truth RL techniques require detailed ad-hoc design of the reward function - in the absence of such ground truth reward functions, such as making a Dog stand on its hind legs or chase its tail, we demonstrate TADPol's superior performance against other externally-supervised techniques such as LIV, CLIP-SimPol, and Text2Reward.

On comparisons with IL: We would like to clarify that we wholly support the development of imitation learning as a field; it is certainly an exciting, interesting, and valuable direction of study. However, we are unable to apply imitation learning techniques for the Dog environment, as there are no ground truth demonstrations for the Dog tasks (such as for chasing its tail or standing on its hind legs). In these cases, without an appropriate dataset of expert demonstrations, imitation learning cannot be performed and compared against.

We believe that when expert demonstration datasets are available, prescribing imitation learning as a solution is appropriate and desirable. We are also interested in further developments in this field, and agree that it has a bright future. On the other hand, RL approaches are complementary to this direction, as they are able to be applied when demonstrations are scarce or nonexistent; in such cases, we demonstrate that TADPol can be applied to learn novel behaviors conditioned flexibly on natural language.

[1] Yi Zhao et al. Simplified Temporal Consistency Reinforcement Learning. ICML, 2023.

[2] Nicklas Hansen et al. Temporal difference learning for model predictive control. ICML, 2022.

We thank Reviewer gAGE for their thorough review. We seek to address their listed considerations below:

• On comparison with text-conditioned work: we appreciate the reviewer for providing a list of related work. We would like to mention that these works: LangLfP, Text-Conditioned Decision Transformer, and Hiveformer, all require training on datasets of trajectories that have been labeled with natural language. Not only are such labeled datasets expensive to create, involving substantial human effort, but the resulting trained policies do not naturally transfer across environments. Indeed, a labeled dataset of trajectories must be created for each novel environment, and across a large number of behaviors and associated text prompts within the environment. In contrast, TADPol enables the learning of text-conditioned policies irrespective of visual environment, and without requiring any pretraining dataset of demonstrations or labeling whatsoever. We do agree with the high-level sentiment of the reviewer, however, that additional comparisons with benchmarks would strengthen the work. In the updated experiments, the results of which can be found in both the updated manuscript and website, we provide comparisons against CLIP-SimPol (Rocamonde et al. arXiv:2310.12921 [a]), LIV (Ma et al., ICML 2023 [b]), and Text2Reward (Xie et al., arXiv:2309.11489 [c]). Such approaches leverage large-scale pretrained models to generate dense rewards for novel behaviors conditioned on text, and we apply them out-of-the-box to arbitrary RL environments. We demonstrate that in the Dog environment from DeepMind Control Suite that TADPol is able to outperform the other methods in learning both goal-achieving behaviors as well as continuous locomotion behaviors.

• On the sensitivity to noise steps: we agree with the reviewer that the choice of noise step might be sensitive to the performance of TADPol. In the updated experiments for the Dog experiment, we thoroughly investigate design choices surrounding the source noise in Appendix B.3. We explore how the frequency of re-sampling the source noise affects the behavior: having a global source noise for all episodes results in more stable, CLIP-like behavior, having a source noise resampled for every frame results in more a more unstable reward signal and therefore policy, and having a source noise re-sampled for each episode is currently proposed as a happy medium. Furthermore, we demonstrate that choice of noise level also affects the resulting policy; too high a noise level (such as using step 750 out of 1000), and the reward signal is no longer meaningfully understanding alignment with the generated frame. We discover a “sweet spot” for the noise level exists around step 450 out of 1000 that is able to extract meaningful priors from the pretrained diffusion model with respect to the generated frame, while avoiding overcorruption. We visualize the results of these ablation experiments in the updated website link.

• On the $k(t)$ function: for our experiments we use $k(t) = \frac{1}{2} (1 - \prod_{i \leq t} \alpha_i )^2$; however, we note that in our experiments $k(t)$ can be treated as a constant, since we only evaluate with a constant t (where $t=450$ in our experiments). However, we include a $k(t)$ term in the TADPol equation to keep it general, allowing it to flexibly change with respect to $t$; in experiments where $t$ is resampled during training, this term provides the option to flexibly adjust the reward computation accordingly.

• On testing on more advanced environments: we take note of the reviewer’s wish to see experimentation on more advanced environments. We therefore report additional experiments on the Dog locomotion environment from the DeepMind Control Suite. The Dog environment is complex in that it has a high degree of freedom of motion, with no task-specific design decisions, allowing the modeling of very flexible and complex behaviors. We demonstrate that TADPol is able to learn goal-achieving behaviors (such as standing on its hind legs or chasing its tail), as well as continuous locomotion movements (such as walking) conditioned on natural language inputs, and invite the reviewer to look at the updated website for the latest visualization results. The background visual environment of the Dog has been modified to appear more natural, and we take these promising results as positive signals for TADPol’s effectiveness in real-world scenarios.

We thank Reviewer fbDb for their review, and seek to address their listed questions and feedback:

• On writing: we appreciate the reviewer’s encouragement to improve the writing of this paper. We update our draft to demonstrate and analyze new experiments to substantially reinforce the merits of our proposed approach. We look forward to the reviewer’s helpful comments on the latest version of the work to improve the clarity of our paper.

• On baseline experiments: In our updated experiments (Section B in appendix) on the Dog environment, we evaluate TADPol against numerous other reward generation methods. We compare against CLIP-SimPol (Rocamonde et al. arXiv:2310.12921 [a]), LIV (Ma et al., ICML 2023 [b]), and Text2Reward (Xie et al., arXiv:2309.11489 [c]), and demonstrate that TADPol is able to achieve superior performance in not only goal-achieving behaviors but also continuous locomotion.

• On the simplicity of tasks: we agree with the reviewer that the merits of TADPol would be better showcased in complex environments such as DMControl. We therefore provide additional experimental results on the Dog environment, a complex, flexible continuous control environment from the Deepmind Control Suite. This environment is complex in that it has a high degree of freedom of motion, and no task-specific reset conditions, allowing the modeling of very flexible and complex behaviors. We further split our analysis across two types of tasks: goal-achievement and continuous locomotion. For goal-achieving tasks, we demonstrate that TADPol learns to successfully achieve a desired pose, such as "standing", "standing on hind legs", or "chasing its tail". For continuous locomotion, we show how TADPol outperforms other approaches both qualitatively and quantitatively in learning motion.

• On motion and temporal information: as mentioned in Section 4.3 we agree with the reviewer that by generating the reward signal solely from individual frames’ alignment with natural language, a fundamental limitation is that there is indeed no conceptual way to determine temporal-extended properties such as direction or speed. In this work we propose two ways to address this problem; firstly, by proposing and comparing against a text-to-video diffusion model. Text-to-video models would be able to distinguish if the entire rolled-out trajectory indeed moves according to a desired direction or speed specification. However, we discover that current text-to-video models do not outperform text-to-image models under the TADPol framework, which is an interesting future direction to explore. Secondly, as stated in Appendix A.2, we are aware that it is often desirable to learn policies that consider non-visual factors as well, such as minimizing control costs or the energy expended by the agent as it interacts. Such information may not be naturally extractable from visual signals, even if videos are used. In such cases, we believe TADPol can be augmented with other available reward signals that consider other factors beyond the visual domain. In our updated rebuttal experiments, explained in the new Appendix B.4 section, we demonstrate that TADPol can be augmented easily and effectively - when adding a simple direction-agnostic speed reward to the TADPol reward for the prompt of "a dog walking", the resulting learned Dog agent indeed is able to move quickly while moving in a natural walking motion. We believe further investigative efforts into how TADPol can be made aware of and utilize motion, temporal, and non-visual information is interesting future work.

We thank Reviewer mJca for supplying useful feedback in response to our work, as well as relevant prior work, which we address:

• On simulator-rendered videos: we agree with the reviewer that there are limited realistic scenarios that would utilize the output of a simulator-rendered video as a meaningful video, beyond some complex behavior demonstrations that might be too expensive or dangerous to perform and record in real life (e.g. how to evacuate a failing spacecraft, or how to perform a high-stakes surgical procedure). We would like to clarify the motivation of using a simulator to render videos as a natural way to treat policies as a form of visual generative model. By interpreting the policy as a model that can generate coherent images/frames (rendered through an environment), we can connect it with diffusion models, which also generate images but with additional text-conditioning capabilities. TADPol can then be motivated as a way to distill these capabilities, acquired from large-scale data and architecture, from a pretrained diffusion model into a policy, to naturally unlock the learning of text-conditioned behavior. We do so by training the policy to generate rendered frames that align with the text prompt as supervised by the priors of a pre-trained frozen diffusion model. Using a simulator to render videos of a policy is therefore the motivating bridge that connects policy learning with existing large-scale pretrained models, and enables the learning of novel text-conditioned behaviors.

• On comparisons with existing video rendering methods: tied with the previous response on the practical usage of simulator-rendered videos, we agree that our goal is not really to render high-fidelity, natural-looking videos for downstream purposes. Indeed, the visual quality of the policy is naturally limited by the strength of the environmental renderer. Aware of these limitations, and as it is also not our main objective, we therefore do not compare TADPol against video rendering methods. Instead, we believe the contribution of highlighting the inherent video-rendering capabilities of a policy in an environment is valuable as a paradigm shift; it exposes a natural bridge through which benefits of scale can be introduced to reinforcement learning, which has generally been siloed off from the scaling benefits observed in other domains such as vision and language. In this work we demonstrate the benefits of incorporating large-scale data pretraining and architectures into reinforcement learning through TADPol, unlocking the learning of policies that are flexibly and accurately conditioned on natural language inputs.

We thank Reviewer LDs4 for the in-depth, constructive feedback. Below we seek to address the reviewer’s concerns:

• On experimentation: we take note of the reviewer’s wish to see more thorough experimentation across more complex tasks, and present additional results on the Dog locomotion environment from the DeepMind Control Suite. The Dog environment is complex in that it has a high degree of freedom of motion, and no task-specific reset conditions, allowing the modeling of very flexible and complex behaviors. We demonstrate that TADPol is able to learn goal-achieving behaviors (such as standing on its hind legs or chasing its tail), as well as continuous locomotion movements (such as walking) conditioned on natural language inputs, and invite the reviewer to look at the updated website for the latest visualization results.

• On comparison with existing benchmarks: we perform and report additional comparisons against existing approaches that seek to generate dense rewards from natural language specification. In particular, we benchmark our approach against LIV (Ma et al., ICML 2023 [b]), as well as Text2Reward (Xie et al., arXiv:2309.11489 [c]). We demonstrated that using a pre-trained LIV out of the box as a dense reward signal is unable to learn meaningful goal-reaching or continuous locomotion capabilities on the Dog agent. Whereas this may be expected, given that the experiments LIV is trained for are goal-reaching robotics demonstrations, fine-tuning LIV for the Dog task requires explicit ground-truth text-labeling of trajectories, which TADPol does not require. Furthermore, we report comparisons against a reward function generated from text descriptions, as proposed by Text2Reward. In these experiments, we prompt ChatGPT with a similar template as proposed in Text2Reward for Hopper, with the details of the particular Dog agent specified instead of the Hopper agent. We discover that the reward signal generated purely from text and code conditioning is indeed able to promote related behavior in the Dog agent, but at a lower quantitative and quality compared to TADPol despite having access to real-time state signals such as speed and direction.

• On the usage of "diffusion policy": the reviewer is correct that the policy does not synthesize predicted actions using a diffusion process; however, in this work we call our framework a "Text-Aware Diffusion Policy" because it is a policy optimized to respect a text-aware diffusion model. In essence, this policy is a distillation of a pre-trained text-to-image diffusion model, transferring all the priors within the diffusion model relevant to the prompt and visual environment into a policy network. Just as a diffusion model generates images from text prompts, so too does the resulting policy continually "generate" frames (by selecting actions and rendering the results through the environment) conditioned on the input text, in accordance with the text-to-image diffusion model it is distilled from. As it is a diffusion model essentially converted into a policy network, we call it a "Text-Aware Diffusion Policy"; however, we are amenable to clarifying this name with an alternative, such as "Text-Aware Diffusion-Distilled Policy", and are open to further suggestions from the reviewer.

• On video generation and reinforcement learning: we agree with the reviewer that the end goal of TADPol is about achieving good performance in traditional RL tasks, and that video synthesis is not the main contribution or story of the work. However, we would like to defend the perspective and highlight the value in interpreting policies as implicit video-generators through an environment with rendering capabilities. By treating the policy as a model that can generate coherent images/frames (where visual quality is admittedly at the mercy of the environment), it naturally provides a connection with diffusion models which also generate images but with the additional benefit of having excellent text conditioning properties. This shared perspective inspires the TADPol framework, where we seek to distill the priors and text-alignment captured within a pre-trained text-to-image diffusion model, such that the resulting policy also generates frames that are now strongly aligned with human-provided text. Whereas we do not treat policies as video renderers for the sake of using the output videos, it is precisely because we treat policies as a form of video generator that provides a clear motivation and intuition into TADPol. Indeed, it is through this perspective that a path forward to incorporating the benefits of large-scale pretrained data and models into reinforcement learning, which has generally been siloed off from such scale, becomes promising; and in this work we preliminarily demonstrate such benefits, unlocking the learning of policies that are flexibly and accurately conditioned on natural language inputs.