iVideoGPT: Interactive VideoGPTs are Scalable World Models

We sincerely thank Reviewer 6ajb for providing a detailed review and insightful questions. We have responded to common questions in the global response and individual questions below.

1. Motivation of iVideoGPT: Best of both interactivity and scalability

In $\underline{\text{Q1 of global response}}$, we have clarified that iVideoGPT is motivated by two lines of research and aims to achieve both interactivity and scalability.

To specifically answer Reviewer 6ajb's question:

• Interactivity: We define the term "interactive" as that a model can predict future steps incrementally, conditioned on intermediate actions (as defined in Sec 2 of paper). While Dreamer and many classic models do have interactivity, we remark that many advanced video generation models do not. These include well-known Sora and action-conditioned autoregressive models like VideoGPT as shown in Fig 2b of paper.

• Scalability:

  • Dreamer uses an RNN architecture which has weaker scalability (defined and discussed with evidence in global response).
  • Why scalability is important? Pre-training foundation models with large-scale data is a well-established paradigm in many other fields of deep learning (e.g. languages and the closely related field of video generation), providing impressive success, but is under-explored and not yet commonly applied in model-based control.
  • While the reviewer mentioned Mamba, we argue that Mamba's macro architecture is highly akin to Transformers, with interleaved temporal modules (SSM) and temporal-independent MLPs, which is far from Dreamer. To our knowledge, Mamba has not been used in world models and no such Dreamer variants exist in the literature.

2. Tokenization

Context-dynamics decomposition

To support the decomposition property of the tokenization, we visualized the decoder's reconstruction in $\underline{\text{Fig 1 of attached PDF}}$ by removing cross-attention for future frames. We observed that, without cross-attention, the decoder still reconstructs a trajectory moving in the same way as the original, but the context is almost entirely missing.

Computational efficiency

We measure time and memory usage on the same hardware, which is reported in $\underline{\text{Q2 of global response}}$. Despite the complexity of our encoder-decoder structure, it only takes up a little more resources than a vanilla 16x16 per-frame tokenizer and saves a lot during training and generation of the transformer.

Relationship with scaling

Efficient tokenization greatly saves computational comsuption for training and generation (rollouts), allowing us to scale up the model size with fewer costs.

3. Experiments

Our experiments aim to demonstrate that one architecture and pre-trained model can be adapted into various downstream tasks to achieve competitive performance. Although we do not significantly outperform state-of-the-art methods on all tasks, we believe it is a valuable step.

Video prediction baselines

We only compare with MaskViT on action-conditioned BAIR and high-resolution RoboNet settings, since all other models do not report these settings and many strong baselines such as FitVid or MAGVIT do not release official training codes or instructions.

Visual planning performance

We find that mixed results across models in the VP2 benchmark can be primarily attributed to its imperfect built-in reward design. Please refer to Q4 in global response for details.

Significance of MBRL experiments

To better show the statistical significance of our superior performance, we follow the protocol from [1] and report aggregated performance using Inter-Quantile Mean across all 30 runs in $\underline{\text{Fig 5 of attached PDF}}$. It shows that our method significantly outperforms DreamerV3, a well-tuned strong baseline for this setting.

[1] Deep reinforcement learning at the edge of the statistical precipice

Control benefits of scalability

There are two kinds of scaling: model scaling and data scaling. In our experiments, we currently do not find that further model scaling can significantly improve performance on Metaworld tasks (iVideoGPT-436M performs on par with 138M), likely because Metaworld is a visually simple simulation environment. However, as shown in Fig. 6 of the paper, data scaling does provide benefits---pretraining iVideoGPT enhances MBRL performance. In contrast, Dreamer's lack of scalability limits its potential to benefit from pre-training (see Figs 2 & 5 in attached PDF).

4. Others

MBRL training time

Our method in PyTorch takes ~4 hours on 4090 to train for 100k environment steps. Official DreamerV3 in JAX takes ~1 hour. When fairly compared under the same framework, a DreamerV3 implementation in PyTorch takes ~3.5 hours for 100k steps, comparable to ours.

Difference with MaskViT

iVideoGPT and MaskViT are fundamentally different types of generative models. MaskViT is a masked model that generates all video frames simultaneously through masked reconstruction, typically trained for fixed-length videos. In contrast, iVideoGPT is an autoregressive model, allowing flexible video generation of various lengths. Additionally, as discussed in $\underline{\text{Sec 4.1 of paper}}$, MaskViT uses per-frame tokenization and suffers from temporal inconsistencies, while iVideoGPT employs a novel tokenization conditioned on consistent contextual information, contributing to its improved performance.

Clarification on claims

We apologize for the strong statements and will revise them for accuracy:

• "broad world knowledge" => "common knowledge of motions and interactions in various scenes through pre-training on diverse human and robotic manipulation videos."
• "decouples dynamics information from context conditions" => "designed to encourage the decoupling of dynamics information from context conditions."

Dear Reviewer 6ajb,

As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.

Best regards,

Area Chair, NeurIPS 2024

We sincerely thank Reviewer zSnp for the thorough review and valuable questions. We also appreciate the pointed-out typos, which we will correct in a future revision.

W1: Computational efficiency

We apologize for missing a quantitative efficiency analysis. We have reported training/inference time and memory usages with various tokenizers, in $\underline{\text{Q2 of global response}}$. Our proposed compressive tokenization provides significant memory savings during training and faster rollouts during generation.

Discussion on Dreamer

While we have enhanced the scalability of Transformer-based world models through compact tokenization and resource savings, Dreamer with an RNN architecture may still be more efficient computationally. However, as demonstrated in $\underline{\text{Fig 2 of attached PDF}}$, when pre-training Dreamer XL (200M parameters, comparable to iVideoGPT) on the same dataset as us, we find that it has insufficient capacity to support large-scale pre-training, which is crucial for the success of modern foundation models. The results in $\underline{\text{Fig 5 of attached PDF}}$ further prove that Dreamer is unable to benefit from such ineffective pre-training.

W2: Eliminating latent imagination

We do not claim that eliminating latent imagination is a unique feature of iVideoGPT. However, since latent imagination is currently the dominant practice in MBRL (as seen in Dreamer, MuZero, etc.), we believe it is an advantage for iVideoGPT to simply serve as a plug-in replacement of the environment. As discussed in Sec 4.3, this can simplify the design space, reduce implementation complexity, and enhance the practicality of MBRL.

The key to this advantage is a powerful world model. Previous action-conditioned video prediction models like FitVid, without sufficient capacity and pretrained knowledge, can also function similarly to iVideoGPT but can produce more blurring predictions ($\underline{\text{Fig 4 of attached PDF}}$), thus hinders MBRL directly on top of these inaccurate frames ($\underline{\text{Fig 5 of attached PDF}}$). These results may explain why Dreamer employs latent imagination in MBRL. While IRIS shares a similar architecture with iVideoGPT, it lacks efficient tokenization and large-scale pre-training (as discussed below).

W3: Difference with IRIS

We summarize four key differences between our approach and IRIS. Please refer to $\underline{\text{Q3 of global response}}$.

Q1: Conditioning on the context actions

In our implementation, iVideoGPT is not conditioned on context actions. However, it can easily be extended to support this by adding the embedding of context actions to the slot tokens between context frames. We hypothesize that this would not significantly impact performance, as context actions can likely be inferred implicitly from context frames.

Dear Reviewer zSnp,

As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.

Best regards,

Area Chair, NeurIPS 2024

We sincerely appreciate Reviewer yh7R for providing a thorough review, valuable questions, and a positive evaluation of our paper.

Q1: Technical novelty

Discrete tokenization and autoregressive transformer are prevelant in contemporary deep learning, due to their simplicity and generality. iVideoGPT generally shares this architecture with IRIS, but possesses distinguishing features (discussed below in Q2).

Despite adopting a widely used architecture, we believe the most important contribution of our paper lies in advancing a paradigm of pre-training a scalable world model architecture on large-scale data and adapting it into various downstream tasks. This paradigm is well-established in many other fields of deep learning (e.g., languages and the closely related field of video generation) but is not yet commonly applied in MBRL (see $\underline{\text{Q1 of global response}}$ for extended discussion).

Q2: Difference between iVideoGPT and IRIS

We summarize four key features of iVideoGPT that are different from IRIS:

1. Pre-training and fine-tuning paradigm enhancing sample efficiency;
2. Novel and efficient tokenization enabling faster rollouts (~24x, see Q2 in global response);
3. Flexible action-conditioning design;
4. Off-policy MBRL implementation.

All of these features are highly relevant to visual planning and visual MBRL. Please refer to $\underline{\text{Q3 of global response}}$ for a more detailed discussion.

Dear Reviewer yh7R,

As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.

Best regards,

Area Chair, NeurIPS 2024

We would like to sincerely appreciate Reviewer HkK9 for the comprehensive review and insightful questions. We have addressed common questions in the global response and indivual questions below.

Q1: Amount of action data needed

As described in $\underline{\text{Sec 3.2}}$, we do not use any action data during pre-training. Leveraging more accessible, action-free video data reduces the cost of data collection. Although action-free robot videos are still more expensive than Internet videos, there has been significant progress in data sharing (e.g., BridgeData, RH20T, OXE), making robot data increasingly affordable. What we truly need are unified frameworks like iVideoGPT to better utilize these.

Expensive action-conditioned data are only used for downstream tasks. In $\underline{\text{Sec 4.4}}$, we have demonstrated the few-shot adaptation capabilities for both (action-free) video prediction (Fig 7, 8c) and MBRL (Fig 6). In addition, we have also adapted iVideoGPT with 1,000 action-conditioned BAIR trajectories, which achieves 82.3 FVD.

Q2: Contribution of human videos

As detailed in Appendix A.2, we used a mixture of OXE and SthSth for pre-training the tokenizer, as visual diversity is the key to learning context and dynamics information separation. The transformer is pre-trained on OXE only, as we found many SthSth videos with random camera motions are hard to predict. These choices were based on our experience and a comprehensive analysis is expensive and left for future work.

To assess the contribution of human videos, we pretrain iVideoGPT with pure OXE data (taking almost a week) and evaluate on OXE validation data. We observe human videos indeed help pre-training.

Q3: MBRL baseline with FitVid

Although FitVid is originally designed for video prediction and has not been used as world model for MBRL, we have implemented a baseline using FitVid, by replacing iVideoGPT in our implementation. To predict rewards, we add an MLP head on top of FitVid's latent state, parallel to the image decoder.

As shown in $\underline{\text{Fig 5 of attached PDF}}$, MBPO with iVideoGPT outperforms FitVid on 5/6 tasks and performs comparably on the remaining one. We also qualitatively observe that FitVid's imagined trajectories are blurrier (as shown in $\underline{\text{Fig 4 of attached PDF}}$) compared to ours, which hinders its ability to accurately simulate real environments and may hurt MBRL performance.

Q4: Dreamer baseline with pre-training

As elaborated in $\underline{\text{Q1 of global response}}$, Dreamer, with its RNN architecture, lacks the scalability to benefit from pre-training on real-world videos [1]. To empirically validate this, we have used a pre-training algorithm, APV [1], to pre-train DreamerV3-XL (200M parameters, comparable to iVideoGPT) on the same pre-training data as ours, and then finetune it on Metaworld tasks.

The results in $\underline{\text{Fig 5 of attached PDF}}$, indicate that Dreamer almost does not benefit from pre-training. We also visualize DreamerV3's predictions after pre-training in $\underline{\text{Fig 2 of attached PDF}}$, which is of greatly lower quality compared to iVideoGPT, validating the limited capacity of Dreamer architecture.

[1] Reinforcement Learning with Action-Free Pre-Training. ICML 2022.

Q5: Human study

As official pretrained models are not released for most baselines, we are only able to compare iVideoGPT with VideoGPT and MCVD on the action-free BAIR dataset. We generate videos using three models from the test set and ask users to label preferences between two randomly sampled videos, based on the physical naturalness and feasibility of robot-object interactions.

We collect 386 annotations from 9 persons. The results in $\underline{\text{Fig 6 of attached PDF}}$ demonstrate that iVideoGPT is more preferred by human annotators.

Q6: Lack of real-robot experiments

We would like to clarify that, instead of being 'tested primarily on Metaworld in the embodied setup', our experiments in Sec. 4.2 (visual planning) and Sec. 4.3 (visual MBRL), although conducted in simulation, are both embodied low-level control settings.

We apologize that the limited time of rebuttal is insufficient for us to set up a real-robot experiment. We do not claim a contribution that our method is instantly ready for real-robot applications, which is left for future work, but we believe our method can help advance model-based methods in robotics research.

Q7: Visual planning performance

We find that inconsistent performance in the VP2 benchmark can be primarily attributed to inaccurate built-in reward design of this benchmark. Please refer to Q4 in global response for details.

Q8: "Small" evaluation suite

We respectfully disagree with "The evaluation suite is currently very small". In fact, no baselines cover all three problem settings in our paper. We hope additional results provided in this rebuttal can solidify our evaluations.

Q9: Context-dynamics decomposition in tokenization

While we cannot guarantee in neural networks perfect decomposition and no information leakage, we aim to achieve this through a careful tokenizer design: a bottleneck with much fewer tokens compels to only capture necessary dynamics information for future frames and share contextual information with initial frames, to reconstruct raw pixels.

We visualize the decomposition property in $\underline{\text{Fig 1 in attached PDF}}$. The showcases illustrate that, by removing cross-attention from context frames into future frames, the decoder can still reconstruct a trajectory moving the same way as the normally reconstructed one but the context is almost missing.

Dear Reviewer HkK9,

As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.

Best regards,

Area Chair, NeurIPS 2024