World Model on Million-Length Video And Language With Blockwise RingAttention

We thank the reviewer for helpful feedback. We provide detailed answers to the questions below.

Long-video benchmarks

The reviewer requested an evaluation on long-video benchmarks. We evaluated LWM-1M on the Video-MME benchmark, and the results are shown in the table below. LWM-1M achieves the highest performance in its model size category, outperforming similarly sized models and approaching the performance of models ten times larger, and catching up with proprietary models like GPT-4o and Gemini-1.5-Pro.

Comparing with existing open-sourced LLM supporting context length of 128K in needle retrieval tasks

Firstly, this work was completed some time ago, prior to the availability of existing open-source LLMs with a context length of 128K, such as Llama 3 (proof can be given once we may fully de-anonymize our work). We believe our work has inspired subsequent research on scaling context size.

Secondly, at the time of this work, all open-source and proprietary LLMs suffered from the “lost in the middle” issue [1], where models including GPT-3.5, Claude, and Llama performed poorly on needle retrieval tasks. Our naively trained model exhibited the same issue; hence, we proposed a data synthesization technique, which significantly improved our model to achieve near-perfect accuracy on these tasks.

[1] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157-173.

Dear Reviewer cWVz,

Thank you for your positive feedback and for recognizing our paper's contributions.

We agree with the reviewer’s point that comparing our work to recent state-of-the-arts can further strengthen the paper. Following this suggestion, we have updated the paper with in-date baseline comparisons against state-of-the-art open-source models for our multi-needle retrieval task and long video evaluations

Multi-needle Retrieval

We have updated the paper with more results on multi-needle evaluations on recent SoTA open source models (LLaMA-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Mistral-7B-Instruct-v0.3). The results are shown in Table 2. For convenience, we have copied the tables below. To extend these models to 1M context, we use linear positional embedding scaling (8x) for the LLaMA 3.1, and similarly extend the YaRN positional embeddings for Qwen and Mistral. Although these models perform well on lower contexts (32K / 128K), extrapolating to much longer contexts (1M) is much more challenging, seen by the large drop in retrieval performance for these models. We believe this further demonstrates the effectiveness of our training methodology with progressive training, model-based data synthetization, masked sequence packing and ringattention. Our model significantly outperforms these recent state-of-the-art methods at 1M context, achieving 0.67-0.84 accuracy across different settings compared to 0.0-0.32 for baseline models.

Long video evaluation

We evaluated these state-of-the-art models on long-video tasks. The long-video benchmark VideoMME remains challenging—even GPT-4o and Gemini 1.5 Pro achieve less than 70% accuracy. As the long-video results (paper’s table 4 and included in our previous response) show, LWM-1M achieves the highest performance in its model size category, outperforming similarly sized state-of-the-art open-source models like LLaVA-Video and Long-LLaVA. It approaches the performance of models ten times larger and achieves comparable results to leading proprietary models like GPT-4o and Gemini-1.5-Pro.

We hope that these additional experiments address the reviewer’s concerns, and would be happy to further discuss any additional concerns.

Best,

Authors

We thank the reviewer for helpful feedback. We provide detailed answers to the questions below.

Valuable contributions but marginal technical improvements

While we do provide comprehensive training details, these represent careful empirical investigations that advance our understanding of scaling language models to unprecedented context lengths. Similar to how the original GPT papers advanced the field through careful documentation of scaling strategies on Transformers, our work provides crucial methodological insights that enable training of truly long-context models using RingAttention.

While RingAttention provides an important foundation by resolving memory constraints, our work goes significantly beyond this by solving the fundamental challenges of end-to-end training with million-token context windows. This represents a substantial leap forward in practical capability.

This work makes several methodological contributions that collectively enable a new state-of-the-art in long-context modeling:

• An open-source codebase and models that enable reproducibility and accelerate future research in this domain
• Novel AI-based synthetic data generation techniques that demonstrably solve the previously observed "lost in the middle" problem
• A masked packed sequence approach that eliminates cross-sequence attention artifacts, improving model robustness
• A progressive training methodology that, for the first time, makes million-token context windows practically achievable

Each of these contributions represents a significant technical advancement that was necessary to achieve our results. Together, they provide a comprehensive framework for training and deploying truly long-context language models, opening new possibilities for research and applications in this rapidly evolving field.

Can we evaluate understanding of image details by increasing resolutions with lengthy vision tokens?

It's an interesting idea to use more tokens to increase resolutions. It may not be directly applicable to LWM, because LWM was trained with a fixed tokenizer that encodes at a resolution of 256. While one could theoretically split higher-resolution images into multiple 256-resolution segments, this approach might introduce artifacts at segment boundaries and wouldn't necessarily lead to deeper semantic understanding. However, this would be an interesting future research or application.

Long-video benchmarks

The reviewer requested an evaluation on long-video benchmarks. We evaluated LWM-1M on the Video-MME benchmark, which tests video understanding across different duration ranges. Our results demonstrate LWM-1M's strong performance despite its efficient 7B parameter size. Specifically, LWM-1M achieves scores of 63.7% and 60.8% on medium and long videos respectively, outperforming other 7B models by substantial margins (e.g., +12.3% over Long-LLaVA on medium-length videos). While larger 72B models show stronger performance, LWM-1M's results are notable given its 10x smaller parameter count. The model also shows competitive performance compared to recent proprietary models, though we note evaluation conditions may differ. The results show that LWM-1M achieves the highest performance in its model size category, outperforming similarly sized models and approaching the performance of models ten times larger, and catching up with proprietary models like GPT-4o and Gemini-1.5-Pro.

Details on Multi Needle in a Haystack

We built on top of the original Needle in a Haystack problem (see Section 3.3.2 for details), which was to retrieve a single magic number randomly inserted into some long context. Further details on the multi-needle retrieval evaluation was added to Appendix D, and also shown below for convenience.

In the case of single need evaluation, magic numbers are associated with random cities (“The magic number for San Francisco is 2521233” -> “What is the magic number for San Francisco?”). We extend the task by introducing two variables N and R, where N is the number of needles (magic numbers + different cities) randomly inserted into the context, and R is the random subset of magic numbers asked to retrieve (“What are the magic numbers for San Francisco, …, and Bangkok?”). Correctness is computed by extracting out the numbers retrieved for each city and checked with string matching.

Other evaluation data and trainable parameters

For text and image understanding tasks, and short video analysis, we follow the data and evaluation process used in each respective paper. For short and long YouTube video QA, detailed evaluation protocols are described in Appendix G. For needle retrieval tasks, comprehensive evaluation details are provided in Section 3.3.2 and Appendix D.

During training, all parameters are trainable for all stages.

Results of visual understanding are not SOTA compared with up-to-date VLMs

While we acknowledge that our visual understanding results do not match current SOTA VLMs, this is not the primary focus of our work. Our contribution centers on pioneering end-to-end solutions for scaling context sizes up to 1 million tokens on long video and book sequences - a significant advancement that has not been achieved before. The relatively lower performance in visual understanding can be attributed to our deliberate architectural choices: we opted for a simpler backbone model and an older tokenizer to focus on solving the fundamental scaling challenges. This tradeoff allowed us to tackle the more complex problem of extreme sequence length processing. Using better data and more training are all that’s needed to achieve SoTA performance, as shown in the Chameleon paper. Future work can certainly incorporate more advanced visual encoders, more training, and better data to improve performance while building on our scaling innovations.

The paper largely describes training datasets and stages, lacking in-depth ablation studies and analysis. It relies heavily on previously proposed components like blockwise RingAttention, making it more akin to an experimental record than a research paper.

The reviewer questions the contributions of this paper, characterizing it as 'more akin to an experimental record than a research paper.' We respectfully disagree with this assessment.

While we do provide comprehensive training details, these represent careful empirical investigations that advance our understanding of scaling language models to unprecedented context lengths. Similar to how the original GPT papers advanced the field through careful documentation of scaling strategies on Transformers, our work provides crucial methodological insights that enable training of truly long-context models using RingAttention.

While RingAttention provides an important foundation by resolving memory constraints, our work goes significantly beyond this by solving the fundamental challenges of end-to-end training with million-token context windows. This represents a substantial leap forward in practical capability.

This work makes several methodological contributions that collectively enable a new state-of-the-art in long-context modeling:

• An open-source codebase and models that enable reproducibility and accelerate future research in this domain
• Novel model-based synthetic data generation techniques that demonstrably solve the "lost in the middle" attention problem
• A masked packed sequence approach that eliminates cross-sequence attention artifacts, improving model robustness
• A progressive training methodology that, for the first time, makes million-token context windows practically achievable

Each of these contributions represents a significant technical advancement that was necessary to achieve our results. Together, they provide a comprehensive framework for training and deploying truly long-context language models, opening new possibilities for research and applications in this rapidly evolving field.

There is insufficient analysis of the potential synergies or conflicts between text and video in joint training. Do text and video in joint training mutually benefit each other, or do they have a negative influence on one another?

We respectfully disagree with the reviewer's assessment. Our work directly addresses the challenges of text-video interactions in joint training through multiple technical innovations. Specifically:

1. We empirically observed that naive combination indeed leads to negative interference, which motivated our systematic approach to this problem.
2. The masked packed sequence technique we propose is explicitly designed to manage these interactions and reduce negative transfer.
3. Our ablation studies demonstrate that these mechanisms improve performance on both text and video tasks compared to baseline joint training approaches.

While we agree that understanding the fundamental nature of text-video synergies remains an important open research question, our work makes concrete technical progress by establishing effective methods for joint training that mitigate negative interference. This creates a foundation for future research into positive transfer mechanisms.

We thank the reviewer for helpful feedback. We provide detailed answers to the questions below.

Defining or discussing the concept of a "large world model"

Thank you for raising this important point about terminology. We use the term 'Large World Model' to describe our approach for several specific reasons:

1. The term 'World Model' has precedent in the field, starting with Ha & Schmidhuber's seminal work [1] which established that world models should capture environment dynamics and enable prediction. Our model similarly aims to capture and predict both linguistic and visual dynamics of the world through both language and video generation.
2. The qualifier 'Large' specifically refers to our model's extended context window, which enables modeling significantly longer sequences than previous approaches. This capability is crucial because real-world phenomena often unfold over long time horizons with complex dependencies.
3. While we acknowledge that our model does not capture all aspects of the world (and we make no claims of completeness), it advances world modeling capabilities by:
   • Jointly modeling language and video dynamics
   • Handling extended temporal dependencies
   • Enabling predictive modeling over longer horizons

This usage aligns with recent developments in the field, such as OpenAI's Sora being described as a "World Simulator" [2], while being precise about our specific contribution to this line of research.

[1] Ha, David, and Jürgen Schmidhuber. "World Models." NeurIPS 2018

[2] Brooks, Peebles, et al. "Video Generation Models as World Simulators." 2024

Lacks ablation studies on masked sequence packing and data balancing

We thank the reviewer for raising this point about ablation studies. We apologize if this was not clear in our initial presentation, but we have indeed conducted comprehensive ablation studies on masked sequence packing, with the full results presented in Appendix B. Specifically, Table 8 shows the performance comparison with and without masked sequence packing, demonstrating an 10%~30% improvement in multiple QA tasks. Table 10 shows the comparison with and without proper mixing of synthetic and chat data, showing a trade-off between synthetic data and chat data. To address this concern, we have now added explicit references to these ablation studies in Section 4.2 of the main paper and clarified their key findings.

The paper lacks comprehensive comparisons with recent multimodal LLM works.

The reviewer is concerned that this work does not compare with recent multimodal LLM advancements. This work was completed some time ago, and since then, more multimodal long-context LLMs have been developed. Once the anonymous review period concludes, we can provide evidence that this work predates more recent multimodal long-context models, such as Gemini 1.5 Pro.

Nonetheless, we appreciate the reviewer's suggestion and conducted more experiments to compare with recent works. We evaluated LWM-1M on the Video-MME benchmark, which tests video understanding across different duration ranges. Our results demonstrate LWM-1M's strong performance despite its efficient 7B parameter size. Specifically, LWM-1M achieves scores of 63.7% and 60.8% on medium and long videos respectively, outperforming other 7B models by substantial margins (e.g., +12.3% over Long-LLaVA on medium-length videos). While larger 72B models show stronger performance, LWM-1M's results are notable given its 10x smaller parameter count. The model also shows competitive performance compared to recent proprietary models, though we note evaluation conditions may differ. The results show that LWM-1M achieves the highest performance in its model size category, outperforming similarly sized models and approaching the performance of models ten times larger, and catching up with proprietary models like GPT-4o and Gemini-1.5-Pro.

Dear Reviewer QidU,

Thank you for taking the time to review our paper. We believe our rebuttal has addressed the reviewer’s concerns including terminology, ablation studies, and comparative evaluation. We would greatly appreciate your consideration of our response.

Best,

Authors

We thank the reviewer for helpful feedback. We provide detailed answers to the questions below.

How was OpenLlaMA data mix rate determined

Our data mix was primarily determined through iterative experimentation of training a little bit on different mixes (50% text, 25% text, 16% text, 8% text) and evaluating performance on standard language eval benchmarks (e.g. short text in Table 1). In the end, we decided on 16% as a proportion that would still enable faster training on more vision data per batch without a large sacrifice in text accuracy, which was important to retain for later on video-chat finetuning.

Specific strategy about loss balancing across tasks and contexts

We performed loss balancing by separating each language or vision-language task into their own batch elements, as each task could have a very different proportion of vision / language tokens. For example, if random language data (100% tokens are text losses) was packed in the same batch element as a long-video language task (<1% tokens are text losses), averaging together to compute the text loss would give the long-video text loss very little weight. Thus, vision / text tokens losses would first be averaged within each batch element, and then averaged over the batch dimension.

In addition, we add loss reweighting within each batch element to address some loss imbalance that would occur between the same task examples. For example, image-text QA questions might have a single text token output (True / False), while others would have many text tokens (detailed image description). Loss reweighting was done by applying a multiplier that would give each example equal weight, meaning higher per-token weight on shorter sequences, and lower per-token weight on longer ones. This was quite important, especially when using our model for some downstream image QA tasks (e.g. POPE) that had single token True/False outputs.