The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control

Dear Reviewer 6pVJ:

Thank you for your insightful feedback! We are encouraged by your appreciation of the significance and quality of our work. Below, we address your concerns in detail.

Concern 1: Swin-DPM is similar to Diffusion-Forcing

Thank you for raising this point! While Diffusion-Forcing is indeed an impressive approach, there are notable differences between it and our method, Swin-DPM:

• Length of Generated Videos: Diffusion-Forcing struggles with the accumulation error of auto-regressive models, making it challenging to generate videos longer than 10 minutes. In contrast, Swin-DPM can generate stable-quality videos over extended periods, such as half an hour, far beyond the capabilities of Diffusion-Forcing.

• Real-Time Capabilities: Diffusion-Forcing is not designed for real-time applications and does not focus on optimizing latency, video frame production, and consistency within limited memory, computational resources, and time constraints. Swin-DPM, on the other hand, incorporates techniques to significantly reduce latency and improve real-time video generation.

Concern 2: Importance of Data Mixture

We agree that data mixture is important for enhancing the generalization ability of the control model. However, we believe that multiple game datasets are not strictly necessary to achieve this. Our approach shows that a well-designed dataset and careful training can lead to strong generalization capabilities, even with a single game dataset. Below we report the results of two different settings, trained with game data only and trained with real-world data fusing. The results demonstrate that training fusing real-world data can improve generalization on real-world scenes.

Table: Performance Comparison of Training Strategies

In addition, we report the results of varying the rate at which the action label is replaced with NULL, where the User-Study Accuracy is based on a manual inspection of the generated videos, where we counted how many frames we considered correctly aligned with the control inputs. Please note that these results were reproduced from training on a small subset of the full dataset and may not align with the results presented in the paper.

Table: Effect of Replace Rate on Evaluation Metrics

Concern 3: Action Representation

For the main task, it is not strictly necessary to transfer keyboard inputs into text representations. However, we chose to do so primarily for generalization reasons. The generalization of control in out-of-distribution (OOD) scenarios is achieved by altering the text prompts. The model is pretrained using the DiT model and inherits its understanding of text-based commands. By randomly dropping control signals during training, the model learns to manipulate the movements of real-world objects.

We hope to expand this in future work by collecting a broader range of interaction types and using language as a unified protocol to understand and merge them into the model. This will enable the model to generate novel interactions independently. We did not implement this idea in the current work due to the substantial workload involved, but we see it as an important avenue for future research. This future direction is also the reason we converted action labels to text.

We further conducted an experiment to validate the effectiveness of using transfer action labels in the form of one-hot labels instead of text prompts. Due to time constraints during the rebuttal period, the model may not have been fully trained (note that the numerical results may not be aligned with the paper), but the preliminary results suggest that text prompt-style actions outperform one-hot labels across all metrics. Specifically, here User-Study Acc. suggests we manually look into generated videos and count how many percentage of frames we think is correctly aligned with controls.

We hypothesize the following reasons for this result:

• Text Prompts Provide Semantically Rich Representations: Text prompts offer distinct and semantically meaningful representations for actions. In contrast, one-hot labels are less expressive. For example, concepts like "press W for 1 second, then press S for 1 second" or "moving irregularly" are difficult to capture with one-hot labels but can be easily represented with text-based prompts.

• Base DiT Model Familiarity: The base DiT model has been pretrained with text prompt embeddings, and it cooperates well with these embeddings. Since the model is already familiar with text-based representations, it is more effective when using them compared to one-hot labels.

Table: Comparison of Action Spaces on Evaluation Metrics

Concern 4: Swin-DPM and SCM Parameter Effectiveness

Regarding Swin-DPM, the main parameter we focus on is the stride size. We have evaluated the effect of this parameter through ablation experiments, which we have detailed below. For SCM, there are relatively few parameters to tune. It uses a default setting of 4 diffusion steps with no conditional guidance (CFG). Could you please clarify which specific parameters you are referring to? This would help us provide more targeted insights on the parameter effectiveness.

Table: Effect of different strides over VBench Metrics

Concern 5: How Each Component Affects the Performance of Real-World Transfer

We recognize that real-world transfer is an important aspect of our work, and each component of our method contributes in different ways to improving its effectiveness in this regard. We are currently working on additional experiments to further analyze the influence of each component on the real-world transferability. We will include these findings in the revised version of the paper and the rebuttal as soon as possible to provide a more detailed breakdown of how each component affects the performance.

We hope these clarifications address your concerns. Thank you again for your thoughtful feedback and for considering our work.

Dear authors, thank you for the comprehensive rebuttal. The rebuttal addresses a part of my concerns:

• The discussion on the use of textual rather than discrete actions is convincing
• The showcased ablations on data mixture (mixing with real-world data), stride parameter and action representation provide more insights into the work

In their rebuttal, the authors promise additional results on real-world transferability which I'm looking forward to see in this rebuttal period to finalize my recommendation.

I also encourage the authors to consider and comment on my review points regarding a more in-depth and transparent discussion of the limitations and to reformulate wording in the paper accordingly (see my comments in 'Limitations' and 'Paper Formatting Concerns').

Thank you for raising this. We will add a limitation discussion as follows:

Limitations

While The Matrix shows strong potential, there are several areas that remain challenging and should be the focus of future research and development:

1. Physical Implausibilities and Temporal Inconsistencies: While the model generates visually impressive videos, there are occasional physical implausibilities and temporal inconsistencies. These aspects highlight opportunities for enhancing realism in future iterations.

2. Limited Training Scope: Currently, The Matrix has been trained on only two labeled games (Forza Horizon and BMW). While these initial results show great promise, broadening the model’s training across a wider variety of games and environments will be crucial for improving generalization and real-world transferability.

3. Handling of Dynamic Objects: The Matrix performs well with static objects, such as the BMW car, but handling dynamic objects that exhibit complex behaviors remains an ongoing challenge. Addressing this limitation will be essential for expanding the model's capabilities in more dynamic, real-world scenarios.

These limitations reflect important areas for future study and refinement. While The Matrix is a strong proof of concept, overcoming these challenges will enhance its applicability and performance in diverse environments and more complex scenarios.

We sincerely hope that our response addresses all of your concerns. We would be truly encouraged to receive any additional suggestions for further improving this work. Please feel free to let us know if you have any further comments or recommendations!

Dear Reviewer 6pVJ:

Thank you for your valuable feedback and for your interest in this work. We are encouraged to know that our responses have addressed some of your concerns.

Regarding the missing section on "How Each Component Affects the Performance of Real-World Transfer," we apologize for the delay and are now able to provide a detailed numerical analysis. We have been working diligently to incorporate the valuable feedback from all reviewers.

For this analysis, we sampled 100 prompts from both the in-distribution case (Forza-Horizon Scene) and the out-of-distribution case (Real-World Scene). We then generated 96-second videos using these prompts across stage 3 and stage 4, and 6-second videos across stage 1 and stage 2. To assess performance, we measured the FVD, moving-PSNR, and moving-LPIPS scores. These metrics help us understand how each component influences the generalization ability of the model. The results are provided below.

Dear Reviewer 7MCB:

Thank you for your positive feedback and for appreciating our contributions to dataset development, pipeline design, and the qualitative results of our method. Below, we address your concerns in detail.

Concern 1: High-quality rendering as a feature of the base video model

We agree with your assessment that the base model contributes significantly to the rendering quality. However, we have made substantial efforts to ensure that the introduction of Swin-DPM, SCM, and control labels does not degrade visual quality. Generating long videos with stable visual quality remains a significant challenge in the video generation domain, and it is important to note that the base model alone does not possess this capability.

Concern 2: Real-time generation achieved by the consistency model

We apologize for any potential misunderstanding. While SCM utilizes the consistency model as its distillation method, this technique alone would not have achieved the real-time efficiency we observed.

Although Swin-DPM does not improve inference speed at a global level (for instance, a 4-second video still takes approximately 4 minutes to produce), it dramatically reduces latency. Previously, you would have to wait 4 minutes just to see the first frame; now, the 4 minutes are evenly distributed across each video frame, meaning you can see 4 frames (1 second, containing 16 frames) every 25 seconds. This gives the consistency model a near real-time experience, whereas, without this optimization, you would have to wait for several seconds to view a short video.

Additionally, we optimized the SCM framework significantly. In the initial setup, using just the consistency model + Swin-DPM would yield an inference speed of less than 4 fps, due to several factors:

• GPU Utilization: Swin-DPM does not fully utilize the GPU, preventing linear acceleration across multiple GPUs.

• Inefficient CPU Operations: Certain operations within the DiT pipeline (e.g., calculating rope and other constants) are inefficient in real-time scenarios, limiting performance.

• NCCL and PCIe Communication Bandwidth: In non-real-time scenarios, communication occurs at lower frequencies, so bandwidth limitations are less critical. However, in real-time settings, these limitations become significant.

We have addressed these challenges, resulting in significant improvements to real-time inference speed. The code has been open-sourced for several months, and you are welcome to try it yourself. However, due to NeurIPS' rebuttal policy, we cannot provide direct links here, but you can search for it online.

Concern 3: Lack of important features for world generation

We acknowledge that Matrix is still an early exploration in the field of world models. However, we believe it demonstrates important capabilities that suggest the potential for future world-generation models. Specifically:

• Generalization Ability: As reported in Section 4.3 and shown in Figures 7a and A3 in the appendix (if you experience issues viewing the appendix, we recommend using Chrome or Preview on Mac, as we will address this Adobe version conflict in the revision), Matrix allows for rich out-of-distribution generalization. This is due to its pretraining on a large amount of realistic unlabeled data, showing the potential for building general foundational world models through scaling up training and data size.

• Interaction with Objects: Although we are not able to provide visual examples at this time, a small number of object interactions do exist in the Matrix. For example, when a car collides with a tree or brushwood, the tree shows breakage effects; when the car hits a building, it stops; and when transitioning from land to water, the interaction with the ground changes. While we acknowledge that this is not yet a fully generalized interaction across all objects, these examples suggest that the research trend is moving towards the capability of object interaction. We will add this discussion to the paper.

Concern 4: Oversimplification of the action space

We agree that simplifying the action space may sacrifice the precision of movement. However, this trade-off allows the model to generalize better to real-world settings. As explained earlier, generalization of control in out-of-distribution scenarios is achieved by altering text prompts. The model is pretrained from the DiT model and inherits its understanding of text-based commands. By randomly dropping control signals during training, the model learns to manipulate the movements of real-world objects.

We hope to expand this further in future work by collecting additional types of interactions and using language as a unified protocol to represent them, enabling the model to create novel interactions. However, due to the high workload required, we did not implement this idea in the current work. This future direction also contributes to our decision to convert action labels into text.

We further conducted an experiment to validate the effectiveness of using transfer action labels in the form of one-hot labels instead of text prompts. Due to time constraints during the rebuttal period, the model may not have been fully trained (note that the numerical results may not be aligned with the paper), but the preliminary results suggest that text prompt-style actions outperform one-hot labels across all metrics. Specifically, here User-Study Acc. suggesst we mannully look into generated videos and count how many percentage of frames we think is correctly aligned with controls.

We hypothesize the following reasons for this result:

• Text Prompts Provide Semantically Rich Representations: Text prompts offer distinct and semantically meaningful representations for actions. In contrast, one-hot labels are less expressive. For example, concepts like "press W for 1 second, then press S for 1 second" or "moving irregularly" are difficult to capture with one-hot labels but can be easily represented with text-based prompts.

• Base DiT Model Familiarity: The base DiT model has been pretrained with text prompt embeddings, and it cooperates well with these embeddings. Since the model is already familiar with text-based representations, it is more effective when using them compared to one-hot labels.

Table: Comparison of Action Spaces on Evaluation Metrics

Concern 5: Construction of real-world data

Due to the rebuttal policy, we are unable to provide specific examples of the real-world data. However, we are happy to explain how this data is constructed and how it differs from video data used to train the base model. You can get an idea of what the data looks like by searching for keywords like "4K HUD walking in Tokyo, Shibuya Street." These videos share similar movement patterns to the game data, typically recorded from first- or third-person perspectives, with uniform movement speeds and high video quality. Unlike the base model, which uses a wide variety of videos from the internet, our real-world data focuses on walking, driving, and other movement scenarios captured from first- or third-person perspectives.

Concern 6: More cases in Cyberpunk and DROID and their generalization ability

While we are unable to provide additional video cases due to the rebuttal policy, we can address your concerns. In terms of generalization ability, the answer is yes for the Cyberpunk cases, but no for DROID. The DROID scenario is more specialized and narrow, making generalization to other domains more difficult. However, the Cyberpunk dataset generalizes well, as the base model is already familiar with urban scenes.

Concern 7: Why is the action space of Cyberpunk limited to four directions of camera movement and forward motion?

The main reason for limiting the action space is to reduce the complexity of the data. Encoding all possible walking directions and camera movements would result in over 18 control signals, increasing the amount of data required for training. This would place a heavy burden on data collection, filtering, and balancing. We experimented with this approach but abandoned it due to the extreme workload. While the current action space can logically produce similar movement effects to the 18 possible control signals, we decided to keep the model simple for this work.

Concern 8: Does the model support joint training on all game data with a uniform action representation?

Yes, the model does support joint training with a uniform action representation across different games. However, since we only collected data for two games, we did not implement this feature in the current work. The action space design, where all actions are first converted into text, was developed with this capability in mind, as previously explained.

Below, we provide the training results for first training the Forza Horizon 5 scenario alone, followed by joint training on both datasets. The results demonstrate similar performance in terms of visual quality and movement fidelity.

Table: Evaluation Metrics for Joint Training on Different Scenarios

We hope these clarifications address your concerns. Thank you once again for your thoughtful feedback and for considering our work.

I appreciate the authors for their detailed rebuttal. I agree that text can act as a more uniform action condition signal than one-hot action vectors, and I thank the authors for sharing their results and insights. While the lack of method novelty and key technical contribution is still a weakness of this work, I believe the proposed whole pipeline for world generation, which consists of dataset curation, pretraining/fine-tuning, and technical details for high-quality and efficient rendering, outweighs the weakness and is valuable to the community.

However, I find this project has only open-sourced the code for part of the whole pipeline. I hope the authors are being honest here and will release the code of the whole pipeline, including both data collection and model training in the future for better reproducibility, especially the data curation from the game engine, which could benefit many tasks besides video world generation.

Dear Reviewer jKm1:

Thank you so much for your positive feedback and for recognizing our contributions in systematic framework design, dataset collection, and data pipeline establishment, as well as the overall performance of our approach. Below, we address your concerns in detail.

Concern 1: Core task is generating virtual worlds, which could be easily produced using traditional 3D engines.

We appreciate your insightful assessment of the core task and the challenge of generalization in world models. Before addressing the generalization ability of our approach, we’d like to first defend the usage of world models from a few perspectives.

• Why world models are useful, even if they only do what traditional 3D engines can also do:

Perspective 1: Scaling up 3D engines with data is difficult. Imagine you are tasked with creating a sim2real simulation for a company that has built a new factory in a new location. To simulate this scenario, you might need to:

1. Collect numerous videos of the factory, as we did in the DROID dataset, and use them to train a world model, which requires minimal human labor.

2. Alternatively, you could recruit several CG engineers to collect 3D mesh data and develop an entire new 3D environment, which would require substantial time and debugging. While current 3D GS methods can help accelerate the second approach, they still require heavy intervention from experienced CG engineers, which is both inconvenient and expensive compared to the first method. While CG-based methods dominate today, we believe that world models can offer a more efficient, data-driven alternative for creating sim2real simulations.

Perspective 2: 3D engines cannot conveniently provide gradients for downstream tasks. Although techniques like differentiable 3D rendering exist, traditional 3D game engines generally struggle to provide gradient information for downstream tasks. As a result, neural network tasks often require methods like reinforcement learning or Q-learning to handle the absence of gradients. World models, however, can provide gradients directly, making them much easier to use in such scenarios. We discuss this aspect in detail in Section A of the supplementary materials.

• Generalization ability of our method (differentiating it from traditional 3D engines): As reported in Section 4.3 and shown in Figures 7a and A3 in the appendix (if you encounter any issues viewing the appendix, we suggest using Chrome or Preview on Mac, as we will address this Adobe version conflict in the revision), our model demonstrates strong out-of-distribution generalization. By pretraining on a large amount of realistic unlabeled data, our model can generalize to unseen scenarios, which is a key advantage over traditional 3D game engines.

Concern 2: Visual diversity of the generated world.

For the visual diversity aspect, we refer you to our earlier response regarding the generalization ability of our model. The diversity is a direct result of the pretraining on a wide range of real-world data, which enables our model to generate highly varied virtual worlds.

Concern 3: Generated world coherence over time.

Thank you for raising this critical point! Currently, we sacrifice long-term coherence for faster rendering speed. Specifically, we removed long-duration self-attention and use a small history cache. By increasing the range of the self-attention module or expanding the history cache, we can significantly improve consistency, though this will come at the cost of slower processing speed. Currently, we set a stride parameter for the Swin-DPM to trade off between speed and coherence. Stride=1 is used for scenarios pursuing extreme real-time experience ,while larger strides take care of coherence over longer time. We’ve conducted numerical experiments to validate this trade-off, and the results show that increasing the attention range through increasing the Swin-DPM stride size does indeed improve coherence.

Table: Effect of different strides to improve coherence (VBench Metrics)

To collect those metrics, we select 100 prompts randomly sampled from the testing dataset, and generate 768 frames (48s) each from those prompts. We use the custom_input mode in VBench to evaluate those results under the above seven metrics.

Concern 4: Will the quality degrade over time?

To evaluate long-term performance, we measured the FVD score over time, and the results are provided below. Our model demonstrates consistent and stable performance across long durations, with no significant degradation in quality. Although the first 4 seconds show substantial improvement, the quality remains stable throughout the remaining duration, with no noticeable decline.

Table: Evaluation Metrics Across Different Time Durations of The-Matrix

We also further evaluate the performance over time using the imaging qualitymetric from the VBench. We sample 100 prompts from the in-distribution case (Forza-Horizon Scene) and out-of-distribution case (Real-World Scene), and generate videos of 96 seconds with them. We then measure the imaging quality metric of those videos at different times as follows:

Table: Performance across different time windows and environments

While OOD cases witness a slight degradation of quality, in-distribution cases are much stable and barely lose quality as time grows. Both of the cases can still maintain no-collapse visual quality as time grows.

Concern 5: Human labor used in data annotation.

Almost the entire data collection process, except for the Cyberpunk dataset, was automated, as detailed in Section B of the appendix. However, some limited human labor was involved to maintain the realism of the training data.

• For non-Cyberpunk data (e.g., DROID), we employed 10 annotators to review video clips and ensure they contained valid motion scenes (e.g., walking, running, juggling in the wild). This process took approximately 10 days to complete.

• For Cyberpunk 2077 data, we recruited 20 human annotators to play the game and collect data. The collection process lasted 24 days (6 hours per day), during which approximately 2,880 hours of data was collected. This data was split into around 1.8 million 6-second video clips paired with control signals. We developed a visualization tool to assist annotators in making their keyboard inputs more uniformly distributed and used algorithms to filter out low-quality data (e.g., data where the player was stuck or made incorrect actions). The final dataset consisted of about 1 million valid data pairs.

We hope this clarifies the data annotation process. Please let us know if you need further information.

We sincerely appreciate your thoughtful feedback and hope that our revisions address your concerns. Thank you again for your consideration.

Dear Reviewer m9ww:

Thank you for your positive feedback. We are especially encouraged by your appreciation of the GameData platform, Swin-DPM, and the visual results of our methods. Below, we address your concerns in detail.

Concern 1: Contribution of SCM and Swin-DPM to real-time efficiency

We apologize for any potential misunderstanding regarding the contribution of SCM and Swin-DPM to real-time efficiency. While SCM uses consistency modeling as its distillation method, this technique alone would not be sufficient to achieve the final level of efficiency.

Swin-DPM does not directly improve the inference speed of the videos from a global level (for example, a 4-second video still takes around 4 minutes to produce). However, it drastically decreases the latency. Previously, you would have had to wait 4 minutes to see just the first frame. Now, the 4 minutes are evenly distributed across each video frame, allowing you to see 4 frames (1 second of video, which contains 16 frames) every 25 seconds. This means the consistency model provides you with a near real-time experience, rather than the long wait that would be required to view several seconds of video in non-real-time settings.

Furthermore, we have significantly optimized the SCM framework. In the initial setting, using only the consistency model and Swin-DPM would result in an inference speed of less than 4 fps, due to the following reasons:

• GPU Utilization: Swin-DPM does not fully utilize the GPU, which limits the parallel attention computation, preventing linear acceleration across multiple GPUs.

• Inefficient CPU Operations: There are several inefficient CPU operations within the DiT pipeline (e.g., computing rope and other constants) that are typically not a concern in non-real-time scenarios but drastically limit real-time efficiency.

• NCCL and PCIe Communication Bandwidth: In non-real-time scenarios, communication happens at a relatively low frequency, so bandwidth limitations are not significant. However, in our real-time setting, bandwidth becomes a critical issue.

We have addressed all of these challenges, resulting in significant acceleration for real-time inference. All the code has been open-sourced for several months, and you can try it yourself if you're interested! However, due to NeurIPS' policy, we cannot provide direct links here. You can easily find the code by searching for it on Google.

Concern 2: Hyperparameter sensitivity

Thank you for raising this point! We conducted a numerical ablation study on the probability of the "replace label" (q), and the results are shown below. Additionally, the User-Study Accuracy is based on a manual inspection of the generated videos, where we counted how many frames we considered correctly aligned with the control inputs.

Table: Effect of Replace Rate on Evaluation Metrics

We believe the results demonstrate that while the hyperparameter is sensitive, we have identified an optimal range that provides the best trade-off between accuracy and robustness. We will include more details in the revised manuscript to clarify this.

Concern 3: Stride choices

Yes, we did experiment with larger stride sizes, and we observed that they can improve consistency and coherence in generation. However, this comes at the cost of increased computational burden. Larger strides result in larger feature sizes that need to be stored on the GPU, and the square rate of self-attention operations increases accordingly. Given these trade-offs, we chose a stride of 1 for the best efficiency, balancing computational load and performance. Attached is a numerical evaluation of stride size vs. video coherence using VBench (time cost increasing could be viewed as linear growth to the stride size). To collect those metrics, we select 100 prompts randomly sampled from the testing dataset, and generate 768 frames (48s) each from those prompts. We use the custom_input mode in VBench to evaluate those results under the above seven metrics.

Table: Effect of different strides to improve coherence (VBench Metrics)

Concern 4: Improving manuscript writing and proofreading

Thank you for pointing this out! We will carefully polish the paper during the revision process to improve clarity and readability.

We sincerely hope that these clarifications address your concerns. Thank you again for your thoughtful feedback and for considering our work.

Dear Reviewer m9ww,

Thank you for your insightful question! We appreciate the opportunity to clarify this aspect of our work. There are indeed considerations from both perspectives here, and we hope to provide a clearer explanation.

First, the simplification does not actually occur when converting the control signal from 'WASD' to text prompts. 'WASD' are simply one-hot labels, conveying no more information than the text prompts themselves. Both 'WASD' and the text prompts vary across frames (each frame has an individual, different prompt 'the care is driving xxx' to describe the action of the exact current frame), so from an informational perspective, they are essentially the same thing.

The true simplification comes from the decision to adopt only 'WASD' as the action space. We do collect more detailed data, such as the car's speed along the x, y, and z axes, the angular velocity of the tires, and other state parameters. However, we ultimately chose not to use these values in the control for the following reasons:

1. Causality and Simplicity: The parameters we collect (e.g., speed, tire angles, accelerations) are outcomes of the 'WASD' controls. From the perspective of causal reasoning, 'WASD' provides the simplest and most direct signal. The main goal was to focus on the cause of movement, and 'WASD' serves as the clearest, most efficient representation of this cause. Whether we describe it as "pressing 'W'" or "the car moving forward," both convey the same information at a logical level.

2. User Experience: One of our main design goals is to provide a user-friendly interaction that mirrors traditional game controls. Asking users to provide detailed per-frame data such as speed or other state parameters would complicate the process and take away from the experience. Simplifying the action space to 'WASD' makes the system more accessible, allowing users to interact intuitively with the model. Requiring additional data would likely overwhelm users, and simplifying control to something familiar, like 'WASD', helps achieve a more enjoyable user experience.

3. Game Limitations: In the case of some games like Cyberpunk, we cannot access detailed movement data due to the game's anti-cheat mechanisms. As such, we are limited to tracking 'WASD' and mouse movements. Additionally, obtaining detailed movement statistics like speed and acceleration would require modifying the game, which is often technically challenging and potentially against the game’s terms of service. This is why we did not pursue this route in the current implementation.

However, in other environments such as droid simulations, we do use precise angular movements as input (as shown in the appendix). In these cases, we chose not to convert those precise movements into text prompts because we could not find appropriate textual descriptions for the complex movements of all seven motors involved.

We hope this explanation provides a clearer understanding of our approach. There are valid reasons for choosing 'WASD' as the control space, but we also recognize the trade-offs and the simplifications involved. If you have any further questions or suggestions, we would be happy to clarify further.

Dear Reviewer TvGJ:

Thank you for your valuable feedback and suggestions! We greatly appreciate your recognition of the novelty, significance, and originality of our work. Below, we address your concerns in detail.

Concern 1: Why is quantitative analysis against previous methods difficult?

We understand your point about the importance of including quantitative comparisons with previous methods. While we agree that such comparisons would strengthen the paper, we faced several challenges that prevented us from including them in the main paper. However, during this rebuttal period, we have conducted a comparison with Diamond. We hope this addition will help clarify the challenges we encountered.

• Reason 1: Limited Open-Source Options:

Among the competitors listed in Table 2, only Diamond and Oasis offer limited open-source code, both of which were primarily trained on the Minecraft dataset. Other methods do not provide their code or training pipeline, making it difficult to reproduce their results, even using their own datasets.

• Reason 2: Dataset and Architecture Differences:

Another challenge we encountered is the significant difference between the Minecraft dataset (used by Diamond and Oasis) and the datasets we used, such as Forza Horizon and Cyberpunk. This mismatch in data labeling and network architecture made it challenging to apply their models directly to our data without significant modifications. This is why we opted not to include a comparison in the main paper.

Finally, we conducted a comparison between our method and the Diamond world model in the Forza Horizon 5 setting. To ensure a fair comparison, both models were trained from scratch using 50,000 data clips over 4 full epochs. We used the default CSGO settings for Diamond. While Diamond requires additional labels like mouse clicks, weapons, etc., we randomly selected one example from the Diamond training set and inherited all training labels from it, except for the WASD direction label. While this approach might introduce some unintended biases, it was the most practical choice given the constraints.

The results are provided below. Please note that the results may not align exactly with the ones reported in the original paper. Additionally, the User-Study Accuracy is based on a manual inspection of the generated videos, where we counted how many frames we considered correctly aligned with the control inputs.

Tab. Comparason with Diamond on Forza-Horizon 5 Dataset

Concern 2: Degradation of visual quality vs. time

Thank you for your insightful suggestion! We are pleased to provide an FVD vs. time table below to demonstrate the effectiveness of our method in generating long-duration videos. While the initial 4 seconds show significant improvement, the remaining time durations do not exhibit noticeable degradation in quality.

Table: Evaluation Metrics Across Different Time Durations of The-Matrix

We also further evaluate the performance over time using the imaging qualitymetric from the VBench. We sample 100 prompts from the in-distribution case (Forza-Horizon Scene) and out-of-distribution case (Real-World Scene), and generate videos of 96 seconds with them. We then measure the imaging quality metric of those videos at different times as follows:

Table: Performance across different time windows and environments

While OOD cases witness a slight degradation of quality, in-distribution cases are much stable and barely lose quality as time grows. Both of the cases can still maintain no-collapse visual quality as time grows.

Concern 3: High computation cost (8 A100 GPUs)

We are glad to report that our method now runs smoothly on 8 or even 4 L40 GPUs, utilizing a PCIe 4.0 connection with 64GB/s throughput—no need for NCCL. The rendering speed is over 32 FPS, which significantly reduces the computational cost to approximately $0.5 per hour for users. This represents a giant leap in both cost and speed compared to other world models based on diffusion methods!

Furthermore, the entire codebase (which utilizes VNC to broadcast video frames from a remote server) has been open-sourced on GitHub for several months. It has already supported various startups working on world model products. The documentation includes detailed tutorials on how to use the code, and it is easy to set up. Unfortunately, due to NeurIPS policy restrictions, we cannot provide direct links to the code and supported products in this rebuttal, but we encourage you to explore these resources if you're interested. Attached is the GPU running time report of the Matrix DiT backbone under different settings.

Concern 4: Typos

Thank you for your careful proofreading! We will address these typos in the revised version of the paper.

Concern 5: Use of the word "foundational"

Thank you for pointing that out! We will replace "foundational" with "novel" to improve the coherence of the context.

Concern 6: Document Viewing Issues

We apologize for the inconvenience you experienced. This may have been due to a version conflict in Adobe Acrobat. We will ensure this issue is resolved in the revision.

We sincerely hope these clarifications and additions address your concerns. Thank you again for your thoughtful feedback, and we appreciate your consideration of our work.

Thank you for responding to my review and addressing my criticisms. I appreciate the additional Diamond baseline and improvements additional testing on various GPU setups to illustrate model efficiency in terms of runtime compute (though this was a more minor criticism). Although, in the future it would be good to have a more extensive comparison to Diamond (or other method) in the multiple environments presented in this paper. I have increased my score to reflect the authors' feedback.