Seer: Language Instructed Video Prediction with Latent Diffusion Models

Weakness 2: The writing could be improved for better clarity. Additionally, the positioning of figures could be optimized for easier readability; consider rearranging Figures 3 and 4.

Thank you for your valuable feedback. In response to your concerns, which were also raised by Reviewer shhp and Reviewer xJcW regarding the clarity of our paper, we have undertaken comprehensive revisions to enhance readability on four folds:

1. We have streamlined our motivation section, minimizing detailed discussions on robot learning, and have refocused the introduction to underscore the main contributions of Seer.
2. The mistaken claim that text-conditioned video prediction is more challenging than text-to-video generation has been clarified in the related work section.
3. To improve the understanding of our method, we have highlighted key components of Seer in the method figure and reorganized the flowchart for better clarity.
4. Figures have been strategically repositioned to enhance overall paper readability.

These revisions have been diligently incorporated into our paper to address the concerns raised, with the aim of significantly improving the overall readability of our work.

Weakness 3: It would better to enhance the paper to include analysis of failure cases and generated video examples for a more comprehensive evaluation.

Thank you for your constructive suggestion. We have expanded our qualitative analysis of failure cases, incorporating a dedicated section in Appendix D.7 of our revised paper. Figure 18 in the revised paper showcases instances where Seer struggles to capture appearance consistencies, such as the color of the card in the previous frame and the cover of the book in the reference environment. These challenges are attributed to the difficulty Seer faces in forecasting newly occurring concepts and maintaining temporal consistency within complex backgrounds. In the context of the Epic-Kitchens-100 dataset, Seer tends to predict camera pose movements and generate novel views of the environment, rather than accurately learning human motion. This difficulty arises due to the prevalent scene transitions in Epic-Kitchens. Further insights into these failure cases are discussed in detail in the new Appendix D.7 section.

Weakness 1: While the paper repeatedly claims to have designed a sample and computation-efficient model, the only comparison provided in the appendix is in Table 8, focusing on temporal attention. It would be beneficial to include more computational comparisons to validate this claim.

Thank you for your valuable suggestion. We have included an additional comparison involving our ablated setting and another baseline method, VideoFusion * $1$ *, which shares a network design based on the Stable Diffusion U-Net. Notably, in our proposed Seer setting, the 2D spatial layers of the U-Net are frozen during fine-tuning. In contrast, the ablated setting keeps all 2D spatial layers trainable. We assessed GPU memory consumption and training time for different models, utilizing a single 80GB NVIDIA A800 GPU during the training phase. The results in Appendix Table 8 of the revised paper, reveal that the proposed Seer setting with frozen 2D layers exhibits approximately half the GPU memory consumption (24.7GB) compared to Seer without frozen 2D layers (39.2GB) and VideoFusion (45.0GB). To ensure a fair comparison of training speed among various settings on the single A800 GPU, we conducted additional assessments of models with GPU memory usage exceeding 90%, as detailed in Appendix Table 9 of the revised paper. The results in the table reveal a notable advantage in computation efficiency for the proposed Seer setting with frozen 2D layers, exhibiting reduced training time (3.10 sec/iter) compared to Seer without frozen 2D layers (6.89 sec/iter) and VideoFusion (7.63 sec/iter). These findings underscore the heightened computational efficiency of Seer compared to key video baseline models.

Appendix Table 8. Training time (time.) and GPU memory (Mem.) consumption of the models on a 16-frame video

Appendix Table 9. Training time (time.) and GPU memory (Mem.) consumption of the models on a 16-frame video (>= 90% GPU memory usage)

References:

$1$ Zhengxiong Luo, et al. "Videofusion: Decomposed diffusion models for high-quality video generation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10209–10218, 2023.*

Question 6: Why were 2 reference frames used for Something Something-v2, but not for Bridgedata or EpicKitchens100?

This comment is regarding the choice of reference frames in training datasets. The Something Something-v2 (SSv2) dataset, which is concerning real-world human behaviors, contains various unexpected cases in real-world video such as object occlusions, scene changes, and camera view movements. Initially, we were concerned that Seer might face challenges in learning from such intricate scenes when provided with only one reference frame and lacking additional background information. Hence, SSv2 was chosen as our primary training dataset, and we employed the initial setting of using 2 reference frames for generating 12-frame-long videos on SSv2.

To our surprise, Seer demonstrated robust performance even with only one reference frame in the SSv2 dataset in our follow-up test in Appendix Figure 14 of the revised paper. This result prompted us to extend our experiments to the Egocentric human kitchen video dataset EpicKitchens100, adopting the 1 reference frame condition. Remarkably, Seer exhibited competitive performance compared to other baseline methods in this setup.

Question 7: How does such a model perform for long-form generations? Can the final frame(s) of an output sequence be used as a reference to generate subsequent frames using the same text prompt to create a longer video?

Thank you for your valuable suggestion. In this paper, our primary goal is task-level video prediction with language guidance, and most downsampling video clips inherently have sufficient length for task-level behavior performance. Intuitively, appending an additional temporal upsampler network is compatible with most video diffusion models including Seer further aid Seer in expanding video frames.
Moreover, considering the suggestion about using the final frame of the output sequence as a reference for generating the next sequence, this approach could offer the advantage of sampling longer video sequences with the alignment of a long sequence of subinstructions. Therefore, we realize the potential benefits of exploring the expansion of macroscopic-level global instruction to multi-step intermediate guidance via a Large Language Model (LLM) for long-sequence video prediction. This direction is certainly worth further exploration in our future work.

Regarding your suggestion on using the same text for long video generation, we have conducted an additional experiment on long video prediction using a 12-frame SSv2 fine-tuned Seer model. The results of this experiment are presented in Appendix D.6 Figure 19 of the revised paper. These findings demonstrate that using the same sub-instruction to generate the next sequence of output can maintain coherent motion, persisting until the target object is no longer present or the current motion is in a terminated state. While this strategy enables the generation of coherent movements in the output sequence, it is distinguished from the upsampling of video clips along the frame axis. Based on these outcomes, we believe that expanding the generated subinstruction embeddings along the frame axis during the training stage represents a promising direction for enabling Seer to generate longer video frames in multi-steps.

Weakness 4: Seer model finetuned on SSv2 is only able to generate 1 second of video, can enough happen within one second where finegrained details such as subinstructions are necessary. If the authors are downsampling videoclips by sampling frames from the entire video, then Seer is only learning to produce intermediate snapshots of a given natural video.

Thank you for your constructive comment. Many existing approaches, such as Make-A-Video and Video Diffusion Model, extend video length by incorporating Upsampler Networks. However, considering our primary focus on task-level video prediction achieved through downsampling of video clips and emphasizing computation efficiency, we choose not to adopt the Upsampler Network in our work. Furthermore, like most video diffusion models, Seer is compatible with various Upsampler Networks. While we haven't incorporated it in our current work, the integration of Upsampler Networks with Seer could be a feasible direction for future study, particularly when adapting to longer video datasets.

On the other hand, the frame-per-second (FPS) of downsampling in our SSv2 training dataset varies from 1.67 FPS to 12 FPS. Given that most human behaviors in the SSv2 scenario involve one-step motions (e.g., "boiling water"), we have empirically observed that a minimum of 2 frames for a one-second clip is generally sufficient to depict coherent motion in the majority of cases within the SSv2 dataset.

Weakness 5: Additional steps need to be performed to ground generated text-conditioned videos to actual policies; given the motivations, it would be stronger to demonstrate an example (even synthetic) where the videos generated by Seer was able to help policy learning either in a robot, simulated environment, or game.

Thank you for your suggestion. We have incorporated additional visualizations of Seer's performance on robot benchmarks, including MetaWorld, RLBench, and Cliport, in Appendix D.5 of our revised manuscript. In this setup, commencing with the Seer model fine-tuned on SSv2, we have frozen the FStext decomposer and conducted further fine-tuning on a diverse dataset comprising SSv2, Bridgedata, 236 MetaWorld video clips, 785 CliPort video clips, and over 2000 video clips from RLBench. This fine-tuning process spans 80k steps with a reduced learning rate of 4e-5.

The results of these experiments are presented in Appendix Figure 18 of our revised paper, illustrating Seer's adaptability to a multi-task environment that includes human video, robot manipulation, and simulation scenarios. Importantly, Seer maintains robust temporal alignment with task descriptors across these diverse video domains.

Furthermore, to investigate whether Seer can help policy learning, we choose the UniPi $1$ as our baseline in the simulated robotics environment Meta-World $2$ , which generates videos from the initial state and infers actions from the adjacent frames via a pre-trained inverse-dynamics model. Specifically, we distill a policy model from the videos generated by the Seer and the labeled actions from the pretrained inverse-dynamics model. We choose 3 tasks and use 10 in-domain videos for each task to fine-tune the Seer. And we make comparisons between:

a. Distilled policy from the dataset generated by the Fine-tuned Seer model (1000 generated videos for each task).

b. Distilled policy from the 10 in-domain videos for each task.

The results are as follows:

We find that compared to utlize the given 10 in-domain videos to generate policy, fine-tuning the Seer with them and generating more videos can be better, because it can acquire more scalable data, which is of comparable quality. Therefore, the videos generated by Seer can help policy learning in simulated robotics tasks in a way.

$1$ Du, Y., Yang, S., Dai, B., Dai, H., Nachum, O., Tenenbaum, J. B., ... & Abbeel, P. (2023, November). Learning universal policies via text-guided video generation. In Thirty-seventh Conference on Neural Information Processing Systems.

$2$ Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., & Levine, S. (2020, May). Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning (pp. 1094-1100). PMLR.

Weakness 1: The figures were small (to the point of illegibility on printed paper) and often not particularly intuitive to understand. I recommend the authors highlight the components of their method with larger and more intuitive figures (beyond arrows and flow-charts).

We appreciate the reviewer's suggestions regarding the clarity of our paper. After a thorough review, we have carefully examined the figures in the method section. To enhance the emphasis on crucial components of our proposed method, we have made explicit highlights in Figures 4 and 5. We hope these revisions address the concerns regarding clarity.

Weakness 2: The authors generate subinstructions by 1) initializing subinstructions using as the global embeddings and 2) performing cross attention with the global embeddings. Therefore, it appears that subsets of the same information within a global embedding is selected for each subinstruction embedding, similar to performing attention on the global embedding. It is more convinced if the global text was expanded to include intermediate info, such as by thinking "step by step" via a pretrained LLM.

We appreciate the reviewer's insightful feedback and appreciate the thorough suggestions provided. Overall, language instructions can be categorized into macroscopic-level and microscopic-level instructions. Macroscopic-level instructions involve abstract narrations of multi-step tasks, such as "cook a pasta dinner." On the other hand, microscopic-level instructions focus on one-step actions, like "boiling water" within the broader context of "cook a pasta dinner." In response to these valuable suggestions, we emphasize that our primary focus is on microscopic-level action generation. Specifically, the FSText decomposer is designed to transform one-step instructions into sequential fine-grained subinstructions. Through the alignment of visual tokens at each time step, these subinstructions effectively plan a continuous motion trajectory. Consequently, these subinstructions can be considered equivalent to low-level robotic instructions, providing specific values such as "rotate hand by 30 degrees then move forward 1/32 of the distance between the hand and the target object."
Furthermore, we agree with the reviewer's suggestion regarding macroscopic-level global instructions. We acknowledge that these multi-step global instructions can be expanded into intermediate information through a pre-trained Large Language Model (LLM). This intermediate information can then be systematically converted into one-step instructions for each state, ultimately serving as the foundation for generating low-level subinstructions for video prediction. And recent approaches such as SayCan $1$ , have attempted grounding language in robotic tasks by translating high-level abstract knowledge of macroscopic tasks into procedures for performing complex and temporally extended instructions. From the above discussion, we see the value in exploring this direction further in our future work.

References:

[1] Ahn, Michael, et al. "Do as i can, not as i say: Grounding language in robotic affordances." arXiv preprint arXiv:2204.01691 (2022).

Question 3: Additional investigations into what the subinstructions may be encoding in terms of textual information would be helpful. Are there ways to probe, textually, the subinstructions learned to understand what they are encoding?

We appreciate the reviewer's observation and understand the challenges posed by the limitations of CLIP in generating readable textual content from encoded embeddings. In the learning process of FSText decomposer, where subinstructions are acquired from transition distributions along the temporal dimension, we have noted that attempting to generate text-to-image content solely based on subinstructions without temporal awareness results in content that is not easily interpretable by humans.

Consequently, to indirectly explore the semantic expression of subinstructions, we have included visualizations of videos generated with the guidance of constant clones of subinstructions from the third or twelfth frame along the video axis. These visual samples are presented in Figure 9 of Section 5.6 and Figure 16 in Appendix Section D.3. Through these visual representations, we aim to illustrate that subinstructions indeed provide proximate semantic guidance for the motion trajectory at each time step.

Weakness 1: The first few paragraphs of the introduction focus on robot learning, but the paper completely eschews this domain. It would be useful to motivate paper from a different perspective that does not involve robotics.

Thank you for your thoughtful suggestion. We acknowledge that we overemphasize robot learning in our introduction section. The primary goal of this paper is to introduce a text-conditioned video prediction (TVP) model to predict task-level videos based on a specific task descriptor that describes behavioral movements in both the real world and robot environment. We have duly noted your feedback and have revised the motivation in the introduction section to accurately reflect the primary focus of our research.

Weakness 2: Difficulty in comparison to text-to-video generation: One could argue that performing text-to-video generation involves learning both the initial state distribution along with the transition distribution, while prediction involves only learning the transition distribution. Therefore, it is unclear which is harder and it would be useful to avoid making such claims.

Thank you for bringing this to our attention. We acknowledge the concern that claiming text-conditioned video prediction is inherently more challenging than text-to-video generation tasks may not be appropriate. In response to this feedback, we have revised the description in the introduction section of the paper to accurately represent the nature of our task.

Question 3: How does sequence length of the video along with number of steps of prediction affect performance?

Thank you for your comments. We have included additional analysis regarding the correlation between sampling steps and the quality of video predictions, particularly with two sequence lengths (12 and 16) in Appendix section B.3 of the revision. Specifically, we conducted an assessment for both 12-frame and 16-frame video predictions using various DDIM sampling steps (10, 20, 30, 40, 50, 60 DDIM steps). All generated outputs were sampled using a 12-frame SSv2 fine-tuned model. The comparative results are illustrated in Appendix Figure 10 of the revision and the following table. Notably, the 16-frame curve experiences a steeper decline from DDIM steps 10 to 20 compared to the 12-frame curve. As both curves progress beyond DDIM step 30, they tend to stabilize and show marginal gains. These results collectively demonstrate that increasing DDIM sampling steps notably improves video quality with longer video sequences when the sampling steps are in the lower range (10 to 30 DDIM steps). However, the improvement of generated video is limited as the DDIM steps are increased to larger than 30.

Additionally, the qualitative results of the 16-frame video are illustrated in Appendix Figure 12 of the revision. Although the generated 16-frame video shows a more downgraded quality compared to the 12-frame video, both 16-frame and 12-frame videos can exhibit coherent motion aligned with language instruction. And since 16-frame videos are not the expected sequence length for a 12-frame model, the generation quality of the 16-frame video can be further improved by training a dedicated 16-frame Seer model.

Appendix Figure 10 results

Weakness 1: Did the authors perform any ablation studies that justified the various design choices in FSText.

Thank you for your suggestion. we have included additional ablation results for the FSText network design in Appendix Table 11 of the revision. To ensure a fair comparison within the same network depth, we opted to replace the ablated component with a Cross-Attention layer. Notably, the Cross-Attention layer plays a pivotal role in the FSText decomposer network for global instruction projection, and our experimentation revealed that removing this layer induces a vanishing gradient problem during training. Consequently, we did not perform an ablation study on the Cross-Attention layer in the FSText decomposer. As presented in Appendix Table 10, our proposed configuration, featuring appended Text-Sequential-Attention layers and Temporal Attention layers, outperforms the other two settings. This outcome underscores the critical functionality of Text-Sequential-Attention layers and Temporal Attention layers in capturing text contextual information and modeling temporal dependencies, which all contribute to the performance improvement effectively.

Appendix Table 11. Components in FSText Decomposer (SSv2)

Weakness 2: For the ablations, could these be reported on more than one dataset?

Thank you for your constructive advice. We have incorporated additional ablation results in Appendix Table 12 and Table 13, specifically including the BridgeData dataset, in our revision. In these results, we consistently observe performance improvements with Seer across various settings. Notably, similar to the ablation results on SSv2, the additional fine-tuning "cross+temp-attn." leads to significant degradation of Seer performance, as observed in the results presented in Appendix Table 13. These findings underscore the robustness of the Seer design, reaffirming its consistent performance across different datasets.

Appendix Table 12. Ablation results of temporal attention in Inflated 3D U-Net (BridgeData)

Appendix Table 13. Ablation results of fine-tune setting in Inflated 3D U-Net (BridgeData)