Solving New Tasks by Adapting Internet Video Knowledge

On AVDC reproducibility: We reuse the open-source AVDC codebase, with the exact same training hyperparameters and architecture configurations that are used to generate the videos shown in that work. We therefore do not believe that the codebase or implementation constitutes substantial discrepancies that may lead to unfair comparisons with alternative adaptation works. However, the reviewer indeed notes that visual quality is slightly worse than when modeling RGB images directly. We identify two main differences that can potentially cause this effect:

1. Latent modeling: Rather than modeling RGB images directly as in the original AVDC paper, we now use the in-domain model to model latents of images because we are now performing score composition with a latent video diffusion model (AnimateDiff). Because each latent is a rich compression of RGB visuals, each latent bit affects the generation of multiple pixels when decoding back to pixel space. Therefore, whereas slight inaccuracies of modeling RGB pixels (as in the original AVDC paper) may not be as noticeable to the naked human eye, such slight inaccuracies of modeling the latents can result in more noticeable artifacts of the final projected RGB image here.
2. Latent dimensionality: The RGB images AVDC was trained on for MetaWorld were of size [128x128x3] (Table 7 of the AVDC paper). However, the latents utilized by AnimateDiff are of size (64x64x4) dimension. Whereas technically each input image from the original AVDC contains 3 times as much data as each latent, this mismatch in input dimensionality may be a source of modeling discrepancy.

On examination of improvement: As mentioned above, we supply baseline in-domain method performance, and demonstrate that adaptation techniques largely have improvement. We have also clarified on our implementation of baseline methods, which reuses existing open-sourced code and settings, where the only change is what we model as input. In our update draft, we highlight our thorough examination of adaptation techniques for facilitating robot manipulation tasks: we apply them across multiple environments and robot settings (e.g. DeepMind Control, MetaWorld, and iTHOR) as well as task types (continuous control, robotic manipulation, and navigation) and video viewpoints (3rd person, egocentric), evaluate their ability to facilitate text-conditioned generalization to novel tasks unseen during adaptation, and also examine suboptimal data usage, where we show how adapting large-scale video models to in-domain can significantly improve performance for downstream robotic tasks without explicitly requiring expert demonstrations. We also provide evaluations with an additional pretrained video model. Across all these settings, we find that adaptation improves quantitative robotic downstream performance over using baseline approaches such as using in-domain video models or generally-pretrained text-to-video models alone.

We thank Reviewer yUek for their careful review and constructive feedback. We would like to address their concerns and questions below:

On Probabilistic Adaptation performance: We update Table 1 with the addition of two new columns: results for inverse probabilistic adaptation, as well as utilizing the small in-domain model only. Also, all table results now include mean and standard deviation computed from 5 seeds. We have also carefully performed additional hyperparameter sweeps, including testing different values of prior strength as well as hyperparameters of Video-TADPoLe, for probabilistic adaptation, as the reviewer suggested; we still observe that the final policy supervision performance collapses. However, as an approach, we discover that inverted probabilistic adaptation achieves nontrivial non-collapsing performance, and clearly outperforms just using the in-domain model. Furthermore, its performance on Dog substantially improves over using vanilla AnimateDiff. This suggests that this form of adaptation of score composition is still promising and applicable for Humanoid and Dog environments, and is not limited to MetaWorld.

Furthermore, as the reviewer suggested, we provide policy discrimination visualizations of Probabilistic Adaptation and Inverse Probabilistic Adaptation for the best hyperparameter settings found on our updated website, in the Policy Discrimination section. We show that even then, probabilistic adaptation is weaker in distinguishing between expert and suboptimal trajectories both in terms of the magnitude of the difference between rewards calculated using expert and poor videos, across different frames. On the other hand, inverse probabilistic adaptation is much more adept at distinguishing between the trajectories, thus supporting its performance on the Dog environment when applied for policy supervision.

On Baseline Performance: We update Table 2 to include in-domain-only baseline performance. We verify that inverse probabilistic adaptation indeed still improves over utilizing only the in-domain model directly, particularly on unseen tasks such as window-open and window-close. This direct comparison supports our hypothesis that adapting a cheap in-domain model trained on small-scale data with large-scale pretrained video models can help generalization to novel, unseen tasks.

On FVD Score: In the original submission, two tasks were selected just to represent performance on tasks seen and unseen during adaptation, and help provide insight into their updated visual performance. At the reviewers request, we have updated Table 3 to be computed across all 7 seen tasks (for the seen column) and all 7 unseen tasks (for the unseen column). 1000 synthesized videos were utilized to evaluate each (adapted) video model, for each task set (seen/unseen) for a more holistic picture of the trends.

On In-Domain Model Performance: With the addition of the In-Domain Model only results for Table 1 and 2, we demonstrate that across both policy supervision and visual planning, adaptation is more favorable to using the in-domain model alone. Noticeably, in Table 1 we observe that using in-domain alone noticeably underperforms compared to using Inverse Probabilistic Adaptation, and in Table 2 it underperforms both Probabilistic Adaptation and Inverse Probabilistic Adaptation. We also highlight that policy supervision as an approach does not require strong pixel-level generation, in contrast to visual planning.

On Qualitative Examples of Subject Customization for Dog Jumping: We update an additional freeform generation achieved by subject customization in the “Novel Text-Conditioned Generalization” section on our website, in which the video again reflects the natural “jumping” motion as specified by the text prompt.

On "High Variance": here, high variance refers to the policy optimization process. Estimating the gradient estimate for the policy parameters depends on trajectories (and therefore rewards) achieved through stochasticity, such as sampling actions from the policy over many timesteps (and potentially stochastic transition dynamics or reward functions in certain environments). We highlight that although using video models to supervise the learning of policies does not necessarily require high-quality in-domain visual generation ability, by virtue of performing policy learning it may suffer from the known issue of high variance gradient estimates. On the other hand, visual planning does not learn a policy and avoids this pitfall, but benefits from high-quality in-domain video generation ability to generate coherent plans.

Avg Steps to Complete Task: We update our Appendix to include the average number of action steps to complete the task, for all successful executions, in Table A11. Unsuccessful executions are avoided in our aggregates, as they merely time-out. As the reviewer suggests, reporting the number of steps not only expresses the expensiveness of applying adapted video models as visual planners, but also provides a sense of the “expertness” of the video plan. We discover that, of successful trajectories, probabilistic adaptation generally has shorter execution plans.

On AnimateDiff vs. StableVideoDiffusion: We select AnimateDiff for our experiments because it is a powerful, text-conditioned video-generative model that has demonstrated strong zero-shot text-conditioned generalization capabilities. We identify text-conditioning as a way to specify the robotic task/behavior of interest - for example, describing “a robot arm opening a door” versus “a robot arm closing a door”. On the other hand, StableVideoDiffusion has strong motion priors, but the open-sourced version does not support text-conditioning. We are therefore unable to use it to explicitly specify tasks of interest, whether for visual planning or policy supervision, for testing task generalization capabilities after adaptation. We update our work to include using a different text-conditioned video model, AnimateLCM, for visual planning on MetaWorld and report the results in Table A9 of our updated Appendix. We discover that AnimateLCM performance is further improved by inverse probabilistic adaptation, with even better performance than that achieved when using AnimateDiff with inverse probabilistic adaptation. This highlights the general benefits of adaptation techniques, untied to particular backbone video diffusion models.

On Details of Suboptimal Dataset: we have added a brief description of suboptimal dataset generation in the main body, for the readers clarity on our setup in the main text, and kept our link to Appendix A for further details.

We thank Reviewer fa6P for their careful review, and for their thoughtful questions and suggestions for strengthening our work. We provide clarifications and additional additional results below:

Quantitatively Evaluating Novel Text-Conditioned Behaviors: we would like to clarify that in contrast to walking, a ground-truth reward function to evaluate jumping is not provided by the environment. In fact, a main motivation for our work is if leveraging some known demonstrations from the environment (such as walking) can facilitate generalization to novel tasks with no predefined ground-truth reward functions through adaptation (such as jumping). On the other hand, for an environment where we can directly test text-conditioned generalization in a quantitative way, we therefore heavily utilize MetaWorld, where there are a variety of task settings with ground-truth success evaluation. Here, we quantitatively compare the ability of adaptation techniques to generalize to tasks unseen during adaptation in a text-conditioned manner, both through visual planning and policy supervision approaches. Furthermore, in our updated experiments (Appendix G, Table A10) we include an additional task that enables us to quantitatively evaluate task generalization performance for novel tasks unseen during adaptation - navigation in the iTHOR environment. We also present such results in our general response, verifying the efficacy of adapting large-scale pretrained video models for generalizable robotic behavior.

On the “best” method, and main takeaways: In Table 1 we have updated the columns to show mean and std calculated over 5 seeds, and we have since included an additional column, for inverse probabilistic adaptation results. We discover that our proposed inverse probabilistic adaptation achieves reasonable performance on the Dog and Humanoid environments, with significant improvement over using Vanilla AnimateDiff in the Dog experiment. Furthermore, it achieves the best overall policy supervision performance for MetaWorld, and competitive performance for visual planning in MetaWorld, as well as the best performance when using suboptimal data. We believe that as a takeaway, inverse probabilistic adaptation is a safe choice to use for adaptation across multiple environments. However, beyond the “best” overall, we also position our work as a study on the expected adaptation performance given considerations on adaptation data and resources (e.g. when direct finetuning is computationally infeasible, or when only suboptimal demonstrations are accessible). We discover that surprisingly, when only still frames of the subject are available, subject customization can still achieve nontrivial improvements for both dog/humanoid and robotic manipulation settings. As subject customization only adapts to the visual characteristics of the environment, this result may be because the environment dynamics of dog/humanoid are similar to those of natural videos, enabling the reuse of large-scale motion priors trained from internet-scale video datasets without modification.

Explaining visual task planning failures: For failed tasks, after inspecting the executed interactions against the visual plans, we found that they still follow reasonably closely - suggesting that task failure does not stem from an inaccurate inverse dynamics model. Furthermore, for probabilistic adaptation, the planned motions themselves also look reasonably respectful of in-domain dynamics. Rather, in many failed tasks, the video plans simply do not appear to perform the specified task (e.g. the robot arm appears to reach towards the net rather than go low towards the soccer ball - failure case visualizations have been included in our updated website at the very bottom). This suggests that rather than the inverse dynamics, or the in-domain motion modeling, the bottleneck may be the learned text-motion alignment in the utilized backbone video model, and perhaps additional improvements in downstream robotic performance can directly follow from further advancements in the field of text-to-video modeling.

On subject customization for novel dynamics: we follow the reviewer’s suggestion to investigate subject customization when the environment dynamics have been modified, by decreasing the gravity constant in the Humanoid environment. We observe that subject customization is still able to learn to maintain the humanoid to be upright and maintains reasonably high ground-truth reward; however, its walking performance is not as high as under normal gravity conditions. We provide such visualizations, and aggregate quantitative results over 5 seeds on our updated website, where it still seems to take step-like motions, but does not make much distance progress most likely due to the change in environment dynamics. We can also imagine that in the extreme, when gravity does not exist at all, or when gravity is overwhelmingly strong, walking is impossible at all. We believe this result is because for subject customization, the model has simply updated based on the environment's visual characteristics, but preserves the motion module from large-scale pretraining. Such large-scale motion priors may be generally applicable across environments, but may not be as robust if the environment exhibits severe out-of-distribution dynamics from its training data (such as floating). This may explain why subject customization performs well in the regular humanoid/dog environments, but for robotic settings like MetaWorld that may require further understanding of dynamics different from what was seen during pretraining of the large-scale video model, in-domain dynamics supervision approaches such as probabilistic adaptation and its inverse perform much better.

Paper Clarifications: we have made numerous edits to our draft in concordance with the reviewer’s thoughtful suggestions. We update the related work to make clear that inverse probabilistic adaptation is a method we propose, we introduce notation and clarifications for the listed Equations in Section 3.1.1. We also clarify that in Equations 1 and 2, $\epsilon_{\theta}(\cdot)$ denotes the trainable small in-domain model and $\epsilon_{\text{pretrained}}(\cdot)$ denotes the frozen, pretrained video model (AnimateDiff in our work) in section 3.1.3. We also provide more description regarding Video-TADPoLe in Section 3.2.2 and directly reference Appendix B for further details. For direct finetuning, we utilize the same number of in-domain videos for probabilistic adaptation and its inverse to form the small dataset, and we clarify the detailed number of videos used for specific tasks in Section 4.1 “Benchmark” paragraph.

Clarifying “out-of-domain”: We denote all tasks as ‘out of domain’ if no demonstrations or examples of such tasks were provided during the adaptation procedure. For example, in MetaWorld, we provide examples (whether just still frames, or demonstrations) of a certain set of tasks (denoted with asterisks in Table A2) and then apply them to unseen tasks (denoted without asterisks in Table A2). Noticeably, certain tasks involve novel objects or table setups unseen during adaptation such as Window or Drawer, where performance may benefit from adaptation with large-scale video models that have observed similar objects during pretraining.

On Dynamics and Constraints: Subject customization leverages a large-scale pretrained motion prior that is applied to arbitrary environments with arbitrary dynamics; the video model has only been adapted to the specific visual characteristics of the environment (but not its dynamics). Therefore, the dynamics do not have to stay fixed, as any dynamics setting is agnostic to the large-scaled pretrained video model’s motion module. However, we can observe that supplying in-domain dynamics information is useful, evidenced by the performance of probabilistic adaptation and its inverse on MetaWorld. Furthermore, we demonstrate that any dynamics information alone, without requiring them to be expert demonstrations, can already adapt large-scale pretrained video models to become better facilitators for robotic behavior - this is shown through the suboptimal data experiments, where demonstrations largely do not even solve the tasks at all, but plausible dynamics of the robotic arm is communicated. We show how inverse probabilistic adaptation is not largely hindered by such constraints, being able to be applied meaningfully to continuous control (humanoid/dog), robotic manipulation, and the additional navigation tasks (iTHOR), and without assuming expert demonstrations (suboptimal experiments).

Dear Reviewer fa6P,

We gently invite you to review our additional experiments, updated draft, and points of clarification, as your feedback is invaluable for strengthening our work. We hope that we have sufficiently addressed your concerns, and if there remain any further questions we would be more than happy to provide additional explanations.

We appreciate your time and consideration,

The Authors

We thank Reviewer NuKB for their thorough, thoughtful comments, questions, and suggestions. We seek to address the outstanding concerns:

On the fundamental assumption: we would like to clarify that we do not explicitly make this assumption; rather, in our paper we seek to investigate to what extent this is the case. Therefore, beyond just adapting with visual characteristics alone (subject customization), we also study techniques that explicitly model in-domain motions (such as probabilistic adaptation and its inverse). A takeaway is that indeed supervising in-domain motions can give superior performance (e.g. inverse probabilistic adaptation performance in Table 2), but a surprising additional takeaway is that just adapting to visual information such as subject customization alone can have substantial improvement across multiple environments and robotic settings (Table 1 and Table 2). As the large-scale pretrained motion prior is unchanged via subject customization, this highlights the strength of leveraging large-scale pretrained video models for downstream robotic applications, and how benefits can be achieved with cheap adaptation requirements (e.g. just static images). Furthermore, while we demonstrate that adapting video models to in-domain motions can improve downstream performance, we also study to what degree expert motions are necessary or if arbitrary dynamics-informing demonstrations are sufficient. In our Suboptimal Data experiments (End of Section 4.3, Appendix A), we verify the surprising result that any in-domain dynamics information, not necessarily just expert ones, is useful for adapting large-scale pretrained video models for robotic tasks. Therefore, we believe that both in-domain dynamics and visual information are useful for adapting video models for generalizable downstream robotic task performance.

On additional video models: We agree that ablating over video models is an interesting direction. We update our submission to include additional experimental results for visual planning using a different video diffusion model - AnimateLCM [1]. In Table A9, we discover that both Probabilistic Adaptation and its Inverse achieve improved performance over just leveraging vanilla AnimateLCM, with Inverse Probabilistic Adaptation performing the best. This suggests that our adaptation explorations are not tied to a particular choice of large-scale text-to-video model backbone.

On writing: we thank the reviewer for highlighting how our writing can be improved; we have made adjustments to the draft accordingly (e.g. in the example mentioned, we have removed the redundant last sentence of 3.1.1 entirely). We have also incorporated the typos listed further down in the reviewer’s comment. We have now introduced the denoising/score function used in Section 3.1.3 in Section 3.1.1 with consistent notation. Edits to the draft are shown in blue font.

On the availability of in-domain video demonstrations: we agree that for settings where demonstrations can be easily achieved, imitation learning such as Diffusion Policy can be explicitly applied. However, there may exist settings where generating in-domain video demonstrations is difficult - such as explicitly controlling the DeepMind Control Suite Dog to jump. Moreso, rather than just difficulty as a consideration alone, imitation learning requires the collection of explicit demonstrations for each new task of interest; even if each individual task is not difficult to collect data for, this can quickly become intractable if the set of tasks of interest is large. We therefore are interested in how large-scale pretraining, such as leveraging large-scale pretrained video models trained on internet-scale data, can facilitate generalization to novel tasks without explicitly seeing examples of such tasks (in which case imitation learning can be directly applied). Rather than just imitating provided demonstrations, we seek to leverage them (via adaptation) to also generalize to novel task behaviors that were not explicitly demonstrated; this is more tractable and feasible than collecting new data for imitation learning for every new task of interest. Furthermore, we agree that optimization and compute cost are of interest in this adaptation; we therefore highlight the performance achieved by each adaptation technique while considering the training data and resource requirements they need. We believe our study across these considerations is useful to the resource-constrained practitioner.

[1] Wang et al. AnimateLCM: Computation-Efficient Personalized Style Video Generation without Personalized Video Data. arXiv:2402.00769 2024.

On Real World Experimentation: we agree that real-world robot experiments are interesting, and believe our method has direct implications for actual robotic experiments. At the same time, however, we would like to highlight that our insights and contributions are still meaningful within these simulated robotic settings. We demonstrate that even in simulation, beyond just adapting to visual characteristics, modeling the movement and interaction physics helps for enabling downstream robotic performance (such as via inverse probabilistic adaptation). Furthermore, while we agree that the interaction dynamics between the robot and environment will be key in adaptation for real robotic experiments, we still expect to adapt to the appearance of the robot arm and its environment, as we do not assume such data is present during large-scale pretraining of the backbone text-to-video model.

On data budget: we agree that the design decisions should be more standardized, and have updated our results in the paper to reflect this. In our updated results, visualized in Table 1, we have standardized the demonstrations for continuous control (e.g. Humanoid and Dog) to be 20 for each environment, across all adaptation methods (e.g. 20 still frames for Subject Customization). Prior literature has not proposed a standardized number of demonstrations for the Humanoid and Dog tasks, to our knowledge, but 20 falls within a reasonable range for per-task demonstrations. We are also interested in studying adaptation with a low-data budget (in the extreme case, Subject Customization only uses still-images); requiring fewer in-domain demonstrations lowers the barrier of utilizing general, large-scale pretrained text-to-video models for downstream robotic applications across environments and behaviors of interest.

On Standard Error: in our updated draft, we update the entries of Table 1 to include standard error calculated across 5 seeds for consistency.

On poor in-domain model for humanoid and dog: we have updated Table 1 to include policy supervision performance when using in-domain only and inverse probabilistic adaptation results on the Humanoid and Dog environments. We therefore empirically verify that in-domain performance achieves poor performance, suggesting its inability to capture the breadth of the complex dynamics within its limited capacity. Both the Humanoid and Dog environments are well-known to have complex transition dynamics in comparison to simpler environments such as Hopper/Walker/Cheetah, or the fixed robot arm in MetaWorld. Furthermore, our updated inverse probabilistic adaptation results demonstrate non-collapsed performance, and clearly outperforms using the in-domain model only. Its performance on Dog also substantially improves over using vanilla AnimateDiff. This suggests that this form of adaptation of performing score composition is still promising and applicable for Humanoid and Dog environments, and is not limited to MetaWorld.

On suboptimal dog jumping behavior in Figure 3: we agree that the video model itself only appears to generate a partially reasonable sequence for a dog jumping. We hypothesize this stems from the fundamental capabilities of the large-scale text-to-video model; and that adaptation has still demonstrated benefits in translating the beliefs of a large-scale text-conditioned video model into something the agent can achieve within the environments domain (e.g. the agent still does seem to be following the video model’s belief after adaptation, even if the video model’s belief is imperfect). We therefore believe that with advancements in large-scale text-to-video models, both in their natural motion priors as well as their ability to align them with text specifications, the accurate completion of downstream robotic tasks should directly improve as well via adaptation.

On tasks with zero success rates: we include such tasks for complete transparency on the limitations of our investigated adaptation approaches; if we were to remove them and only show successful tasks, it may mislead the reader into believing all MetaWorld tasks could be solved through some form of adaptation. We agree that the performance for such experiments could be improved with access to in-domain demonstrations - and if lots of such demonstrations are assumed, then other techniques such as imitation learning could be applied. However, our main goal is to study the performance that can be achieved on novel tasks without associated prior demonstrations but through the adaptation of large-scale generally-pretrained text-to-video models. We also investigate if expert demonstrations are necessary and find that surprisingly, large-scale motion priors and text-conditioning capabilities from powerful text-to-video models can still be adapted with suboptimal in-domain data (End of Section 4.3, Appendix A, Table A1) to facilitate successful performance on novel robotic tasks.

On co-training: we thank the reviewer for pointing us to the co-training technique. In the DROID [2] paper, co-training is used to improve the policy robustness in “out-of-distribution” tasks, by mixing the training batch data with both in-domain task demonstrations and a large-scale dataset collected using a specific robot platform in a balanced manner. In our setup, we treat the videos (or images) used for adaptation as in-domain data, however, we do not assume the availability of a large-scale dataset. Instead, we utilize a large-scale pretrained video model for adaptation, which can be interpreted as a summarization of the large-scale dataset it is pretrained on and with powerful capabilities of generating novel samples. This provides us with a more flexible way of leveraging the knowledge stored in the large-scale data for task generalization. On the other hand, when only a large-scale dataset is accessible, co-training can be applied.

On Inverse Probabilistic Adaptation Performance: as discussed above, we have since updated Table 1 to include inverse probabilistic adaptation performance for the DeepMind Control Suite, with results averaged over 5 seeds, and found that it is able to outperform using the in-domain model alone. It also achieves a substantial performance improvement on Dog over using vanilla AnimateDiff.

[2] Khazatsky, Alexander, et al. "DROID: A large-scale in-the-wild robot manipulation dataset." RSS, 2024.

Dear Reviewer NuKB,

We gently invite you to review our additional experiments, updated draft, and points of clarification, as your feedback is invaluable for strengthening our work. We hope that we have sufficiently addressed your concerns, and if there remain any further questions we would be more than happy to provide additional explanations.

We appreciate your time and consideration,

The Authors

On generalization tests for control tasks: In our evaluations (e.g. Table 2, Table 4 and Figure 3), we have provided quantitative and qualitative results for tasks that were unseen during the adaptation process. We consider these tasks to be novel to the pre-trained video models even after adaptation, and evaluations on these tasks remain valid generalization tests for adapted video models. For example, in Figure 3 we have shown a video generated by an adapted video model that faithfully reflects the novel behavior described by the text prompt “a dog jumping”, along with a successful policy rollout. These results have demonstrated that the tested adaptation methods are able to effectively incorporate in-domain information into pretrained video models while maintaining their internet-scale knowledge and generalization capabilities to achieve strong performance on these novel tasks.

On additional experimentation: In Appendix G of our updated draft, we provide additional experiments on 9 object navigation tasks in the iTHOR environment [3], in which a mobile robotic agent is asked to perform egocentric navigation to specified target objects across various scenes. We also present the experimental results regarding navigation success rates in our general response. Both probabilistic adaptation and its inverse outperform in-domain-only baseline by a large margin, which again showcases the effectiveness of adaptation in leveraging web-scale knowledge to facilitate in-domain tasks. We also highlight egocentric navigation as a new task class to test adaptation over, distinct from the previously reported continuous control and robotic manipulation settings, showcasing the flexibility of adaptation across a variety of robotic task types.

[3] Kolve et al. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv:1712.05474, 2017.

We thank Reviewer DMT5 for the helpful comments and suggestions. We update our manuscript with updated details, and also seek to assuage the concerns of the reviewer in our response.

On paper scope: we consider the general scenario of two learning paradigms: visual planning, where an actionable plan, which could be argued as a dynamically updated visual target, is generated; and policy supervision, where the video model is used as a discriminative reward provider without explicitly synthesizing any visual targets at all. The approaches we investigate to adapt the Internet video knowledge are versatile and general, and can be applied in both learning paradigms. Finally, we would like to refer the reviewer to the ICML 2024 position paper Video as the New Language for Real-World Decision Making (Yang et al.) [1] where they outlined multiple challenges when applying video generation as a tool for decision making. Our work makes concrete contributions to address the “dataset limitations” and “limited generalization” abilities of applying video generation for decision-making (Section 5 of [1]). We do so by only leveraging small amounts of in-domain data for adaptation, and utilizing powerful large-scale pretrained video models to demonstrate how adaptation can be successfully across a variety of robotic environments and tasks (continuous control, robotic manipulation, and navigation).

On experimental surprise: we respectfully disagree. Our goal is not to “produce superior images or videos” for arbitrary domains, as suggested by the reviewer. Our goal is to adapt video generative models for particular environments and embodied agents, both of which are highly unlikely to be covered by web-scale pre-training data. It is thus not obvious if and how pre-trained video priors would help. In practice, we observe that: (1) Pre-trained video models failed to generate coherent videos conditioned on initial environmental state; (2) Models trained on in-domain videos that are multiple orders of magnitude smaller can outperform web-scale pre-trained models on solving tasks through visual planning; (3) Different adaptation techniques have significant impact on the task success rates. All of these signals suggest that it is neither obvious that web-scale models will always help visual planning or policy learning, nor it is clear how to properly integrate web video priors.

On the relationship between generation quality and task success: we would like to highlight that we do not assume FVD scores as an effective indicator for task success; we therefore propose using downstream performance on robotic tasks as another important metric to evaluate and understand adapted video generative models beyond simply looking at visual quality metrics as was done in prior works [2]. In fact, by comparing Table 3 and Table 4 in our updated draft, we can observe that in visual planning there is not always a positive correlation between FVD scores and task performance, and good visual quality alone is not sufficient to achieve task success. How well the generated video plan can properly reflect the motion described by the text prompt is equally or even more critical for task solving. Furthermore, beyond visual planning, we investigate how adapted video models can act as policy supervisors that provide zero-shot text-conditioned rewards to inform in-domain policy learning. In this setting, we only utilize video models to perform one-step score prediction in the latent space for reward computation rather than any explicit generation in RGB space. As mentioned in Section 3.2.2, video models do not necessarily need the ability to create high-quality in-domain videos from scratch to behave as effective policy supervisors; they simply need to be able to critique the quality of achieved in-domain frames. Thus, expressing adapted knowledge through rewards may allow the detachment of downstream policy performance from demands on video generation quality.

[1] Yang et al. Video as the New Language for Real-World Decision Making, Position Paper, ICML 2024.

[2] Yang et al. Probabilistic Adaptation of Text-to-Video Models. ICLR, 2024.

Dear Reviewer DMT5,

We gently invite you to review our additional experiments, updated draft, and points of clarification, as your feedback is invaluable for strengthening our work. We hope that we have sufficiently addressed your concerns, and if there remain any further questions we would be more than happy to provide additional explanations.

We appreciate your time and consideration,

The Authors

We would like to provide further clarifications on comprehensive and challenging nature of our experiment setting:

On the evaluated tasks and benchmarks: We select a wide range of tasks and benchmarks, spanning continuous locomotion (on Dog and Humanoid), robotic manipulation (on MetaWorld) and egocentric navigation (on iTHOR), to comprehensively evaluate adaptation techniques for downstream task performance. These benchmarks, including MetaWorld, are widely adopted by previous works in a similar context, and are recognized as meaningful testbeds under equal or less challenging setups compared to ours. For example, AVDC [1] reports visual planning results on MetaWorld having trained an in-domain model across all evaluated tasks, whereas in our work we evaluate a more challenging setup; we evaluate on tasks for which we do not provide demonstration data. Furthermore, we provide experimentation for settings where only suboptimal demonstration data is provided, whereas prior works all assume expert demonstrations. We hope the reviewer can appreciate the breadth of our experimentation, as well as the more challenging nature of our experimental setup compared to prior work in this area.

On the challenges posed in our setups: We would like to highlight that the challenges in our setups are exhibited across different levels. Firstly, the web-scale data that the large video models are pretrained on can be drastically different from the downstream domains during evaluation, and we hope to bridge this gap by adapting very limited domain-specific examples. Secondly, the selected evaluation benchmarks pose complexity and difficulties from different aspects, achieving competitive performance across all benchmarks is challenging. Thirdly, prior work (e.g. [1]) selects both MetaWorld and iTHOR benchmarks as we do. However, unlike their setup which trains and tests on the same set of tasks, we include considerable evaluations performed on tasks that are unseen during adaptation, to further showcase the generalization capabilities. Especially, in MetaWorld experiments (Table 2, Table 4, Table A9, Table A12), we provide results on 7 out of 9 tasks (denoted with no asterisk) that the model has never observed before the evaluation. Furthermore, we perform adaptation with suboptimal in-domain demonstrations but require the adapted video model to act optimally during evaluation (Table A1).

To summarize, our experimental design is largely built upon those adopted by prior works, but poses more challenges to fully demonstrate the effectiveness and generalization capabilities of adaptation techniques in solving domain-specific robotic tasks.

[1] Ko et al. Learning to Act from Actionless Videos through Dense Correspondences. ICLR 2024.