mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models

For mPLUG-Owl3 7B, in stage 1 we used 32 V100 GPUs to train for about 4 days. In stages 1.5 and 2, we used 64 A100 GPUs to train for 12 hours and 25 hours, respectively.

We found that training the visual encoder reduces the training speed to one-third of its original speed. Due to limited training resources, we considered freezing the visual encoder in previous experiments.

In the additional experiment addressing KHjw Q1, we strictly adhered to the setup of LLaVA OneVision and unfroze the visual encoder during the SFT phase. Compared to LLaVA OneVision, mPLUG-Owl3 achieves a significant performance advantage in BLINK (53.8 vs. 48.2). In conclusion, freezing the visual encoder does indeed affect the performance on BLINK, and unfreezing it can resolve the issue.

We aim to develop a unified model capable of handling single-image, multi-image, and video inputs. Consequently, we are considering a joint training approach, which is also used by mainstream methods (e.g., IDEFICS2 [1] and LLaVA-Next-Interleaved [2]).

To investigate the impact of joint training, we present the results of mPLUG-Owl3, which was first trained on single-image data, and then on multi-image and video data.

The results indicate that training on multiple images and videos can cause the model to forget basic visual understanding abilities, which are helpful for solving complex multi-image and video tasks. As such, simply training on single-image data first and then on multi-image video data results in the model being sub-optimal in various scenarios.

Due to the significant resource overhead of exploring data mixing, we are considering assigning more resources to optimizing and validating the design of Hyper Attention. The training strategy is not the primary contribution of this paper, as we mainly followed insights from previous work.

[1] Laurençon, Hugo, et al. "What matters when building vision-language models?." arXiv preprint arXiv:2405.02246 (2024).

[2] Li, Feng, et al. "Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models." arXiv preprint arXiv:2407.07895 (2024).

This proposal is very constructive. Recent mainstream methods [1,2] adopt a splitting strategy for high-resolution images. Hyper Attention allows the mPLUG-Owl3 to split high-resolution images into more sub-images to retain more details. To investigate its impact, we use OCRBench for zero-shot evaluation to explore the performance of mPLUG-Owl3 in various splitting settings.

Experiment setup

We compare with the baselines Including Mantis-Siglip, VILA 1.5 and LLaVA-Next-Interleave.

The splitting strategy divides a high-resolution image into a maximum of NUM_SUB_IMG sub-images, with each sub-image having a resolution of H×W that is compatible with the model's visual encoder. Consequently, the input resolution for the model can be expressed as H×W×NUM_SUB_IMG.

• Mantis-Siglip does not use the splitting strategy, so NUM_SUB_IMG=1.
• VILA 1.5 and LLaVA-Next-Interleave can split an image into a maximum of 4 images, so NUM_SUB_IMG=4.
• We apply NUM_SUB_IMG = [4, 6, 12] for mPLUG-Owl3.

Results and analysis

The results are presented below. It can be observed that:

• When using the same splitting strategy as LLaVA-Next-Interleave, which splits into up to 4 images, mPLUG-Owl3 achieves comparable performance to LLaVA-Next-Interleave and VILA 1.5. However, Mantis-Siglip significantly compresses the resolution of the input image, resulting in a lower score.
• mPLUG-Owl3 can gain an advantage in OCR capabilities by splitting the input into up to 6 images, resulting in a resolution of 384×384×6.
• Additionally, we increase the NUM_SUB_IMG to 12. Specifically, the candidate splitting grids for mPLUG-Owl3 are set to (2,2), (2,6), (2,4), (6,2), (4,1), (2,3), (3,2), (3,4), and (4,3), allowing mPLUG-Owl3 to handle a maximum resolution of 384×384×12. Under this setting, its performance on OCRBench improves further.

[1] Li, Feng, et al. "Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models." arXiv preprint arXiv:2407.07895 (2024).

[2] Lin, Ji, et al. "Vila: On pre-training for visual language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Since mainstream models use private data and do not disclose training details, making fair comparisons with them is very difficult. Consequently, such comparisons do not yield meaningful insights. Given the limited time for rebuttal and the lower training cost of mPLUG-Owl3 compared to mainstream MLLMs, we are considering re-training mPLUG-Owl3 using a fair training setup. To ensure a fair and rigorous comparison, we will adopt exactly the same training settings and data as LLaVA-OneVision to train mPLUG-Owl3. This will allow us to compare mPLUG-Owl3 with LLaVA-OneVision, a strong baseline that uses publicly available data.

The experimental results indicate that:

• mPLUG-Owl3 and LLaVA-OneVision have comparable performance in single image scenarios.
• When the models are faced with multi-image and video tasks, mPLUG-Owl3 outperforms LLaVA-OneVision by 2.7% and 2.8% in average, respectively.

Meanwhile, as shown in Figure 1 (a) and Table 5, mPLUG-Owl3 also demonstrates significant advantages in memory consumption and inference time.

Since mainstream models are trained on different datasets and training setups, making fair comparisons with them is very difficult. Given the limited time for rebuttal and the lower training cost of mPLUG-Owl3 compared to mainstream MLLMs, we are considering re-training mPLUG-Owl3 using a fair training setup.

To ensure a fair and rigorous comparison, we will adopt exactly the same training settings and data as LLaVA-OneVision to train mPLUG-Owl3. This will allow us to compare mPLUG-Owl3 with LLaVA-OneVision, a strong baseline that uses publicly available data.

The experimental results indicate that:

• mPLUG-Owl3 and LLaVA-OneVision have comparable performance in single image scenarios.
• When the models are faced with multi-image and video tasks, mPLUG-Owl3 outperforms LLaVA-OneVision by 2.7% and 2.8% in average, respectively.

Meanwhile, as shown in Figure 1 (a) and Table 5, mPLUG-Owl3 also demonstrates significant advantages in memory consumption and inference time.

The effects of hyper attention discussed in the current manuscript

In Table 5 of the manuscript, we compared the performance differences between Concatenate (which can be regarded as mPLUG-Owl3 without Hyper Attention) and Hyper Attention under the same data and training settings.

The results suggest that model with Hyper Attention has comparable single image performance and better generalization across multiple images. Meanwhile, Hyper Attention can significantly reduce inference time and memory consumption.

Attest of the hyper attention on LLaVA-OneVision training data

In response to the qTJ2 Q2, we conduct model training and result comparison under the LLaVA-OneVision setup. It can also be regarded as an ablation experiment for Hyper Attention, because both mPLUG-Owl3 and LLaVA-OneVision use the same language model and visual encoder.

Dear Reviewer qTJ2,

Thank you for your volunteer review and your suggestions on conducting isolated analyses of Hyper Attention, as well as comparisons between mPLUG-Owl3 and baselines using fair training data.

We have submitted point-by-point responses to your questions and suggestions. We trust that our responses have satisfactorily resolved your concerns. If you still have further queries or issues with our responses, please feel free to continue the discussion. We are more than happy to engage further.

Hyper Attention is not a improved Q-former. And Hyper Attention does not compress visual features:

• Hyper Attention and Q-former belong to different components of MLLM, performing different functions.
  • Q-Former serves as the alignment module in MLLM, with a role similar to that of the projector in LLaVA. Both of them are language-agnostic and independent of the LLM. As illustrated in Figure 2 and Section 2.1, mPLUG-Owl3 uses a simple linear projection to align visual features with the LLM's space and directly feeds them into the Hyper Attention Blocks to avoid compression.
  • Hyper Attention is a multimodal fusion module inside the LLM. As described in Section 2.2, Hyper Attention avoids modeling the interactions of multimodal sequences directly using self-attention. Instead, it employs a parallel approach using both cross-attention and self-attention to model multimodal interaction. Hyper Attention not only increases efficiency but also enhances the model's generalization ability on image-text interleaved data.
• Q-former is a compression module, whereas Hyper Attention is not.
  • Q-former compress the visual tokens by the language-agnostic learnable tokens before feeding them into the language model. Therefore, textual tokens can only see the compressed visual representation, leading to an issue of visual information loss.
  • Hyper Attention allows for direct interaction between all textual tokens and all visual tokens with a causal mask. As a result, every textual token in the LLM can access all the tokens of the visual content preceding it, similar to concatenate-based MLLMs, thus also avoiding information loss.

Therefore, the Hyper Attention Block is not an improved Q-former. This is also reflected in the experimental results. In the comparative experiments of Tables 1, 2, and 3 in the main paper, our method not only significantly outperforms idefics2 (Q-former like model) but also LLaVA-Next-Interleave (linear projection like approaches).

The suggestion of applying a projection to align visual and textual feature

For your suggestion of applying a projection layer, it may be caused by a misunderstanding of the role of Hyper Attention in mPLUG-Owl3. As illustrated in Figure 2, and described in Section 2.1:

• mPLUG-Owl3 uses a linear projection layer to align visual features with the LLM's space.
• mPLUG-Owl3 uses the Hyper Attention layers to perform multimodal fusion, which avoids the loss of visual information.

We hope the clarifies resolve your misunderstanding. And we welcome further discussion.

There are two dimensions for choising the indices of Hyper Attention Layers: one is the distribution of the layers, and the other is the density of layers.

For distribution, uniformly integrating cross attention layers within a language model has been commonly used in previous methods (e.g., Flamingo [1], ELVM [2], IDEFICS [3]). In early experiments, we also tested designs of introduce Hyper-Attention on the front end [1,2,3,4] or back end [28, 29, 30, 31] of the language model. The comparison is presented below:

Both the two setttings yielding suboptimal performance. Therefore, in Table 11, we maintain the strategy consistent with previous methods to more effectively explore the impact of integration strategy.

For density, we explore the effects of different densities, and the experimental results can be categorized into three types: sparse (2 layers), medium (3-4 layers), and dense (5 or more layers). Therefore, we select three sets of indices to represent these types in the manuscript. Below, we present results for more indices:

As discussed in Appendix E, integrating Hyper Attention at medium density strikes a balance between single-image results and generalization capabilities for videos and multiple images. In contrast, using a dense integration approach worsens both performance and generalization, while a sparse approach still shows strong single-image results but lacks generalization on multi-image and video scenarios.

[1] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." Advances in neural information processing systems 35 (2022): 23716-23736.

[2] Chen, Kaibing, et al. "Evlm: An efficient vision-language model for visual understanding." arXiv preprint arXiv:2407.14177 (2024).

[3] Laurençon, Hugo, et al. "Obelics: An open web-scale filtered dataset of interleaved image-text documents." Advances in Neural Information Processing Systems 36 (2024).

A comparison between mPLUG-Owl3, Qwen2VL, and InternVL is not necessary due to the fairness. Below, we provide the detailed reasons.

1. Contemporaneous work: InternVL2 and Qwen2VL are contemporaneous works; InternVL2 was released on July 4th, and Qwen2VL was released on September 12th. Both are considered contemporaneous papers. As stated in the ICLR reviewer guidline, if a paper was published (i.e., at a peer-reviewed venue) on or after July 1, 2024, authors are not required to compare their own work to that paper.
2. The fairness of training data: InternVL2 and Qwen2VL used private annotations and significantly more training data, which makes the comparison unfair.
   • Privated training data. InternVL2 and Qwen2-VL both include in-house training data, which enables them to have significantly stronger performance in specific domains. For example, in their respective technical reports, Qwen2VL and InternVL2 state that their models are pretrained on manually verified OCR annotated data for better OCR ability. However, mPLUG-Owl3 mainly utilizes open-source data, focusing on model innovation and efficiency in handling long visual sequences, without specifically annotating data to enhance capabilities in particular domains.
   • Training date scale. Qwen2-VL uses 1.4 trillion tokens for pretraining, while the pretraining data for mPLUG-Owl3 is less than 2.2% of that. Our work focuses on exploring the design of Hyper Attention to develop a general and efficient MLLM architecture for single-image, multi-image, and video scenarios, rather than chasing leaderboard positions.

Below, we compare mPLUG-Owl3 trained on public accessed dataset LLaVA-OneVision to Qwen2-VL, InternVL2. The comparison results show that:

• In the single image scenario, mPLUG-Owl3 achieves comparable or slightly lower performance than Qwen2VL and InternVL2.
• In multi-image scenarios, mPLUG-Owl3 demonstrates a significant performance advantage.
• In the video domain, mPLUG-Owl3 comprehensively outperforms InternVL2. For long video tasks such as LongVideoBench, mPLUG-Owl3 also surpasses Qwen2VL.

We further investigate the efficiency and performance of Qwen2VL and mPLUG-Owl3 on a single A100 80G through LongVideoBench. Thanks to Hyper Attention, mPLUG-Owl3 not only outperforms Qwen2VL at various frame rate settings but also has significant advantages in memory usage and inference time.

Dear Reviewer RTBV,

Thank you for your volunteer review and comments regarding the comparison with strong baselines, the confusion about the difference between Hyper Attention and the Q-former, and the strategy for layer indices of Hyper Attention.

We have submitted point-by-point responses to your questions and concerns. We trust that our responses have satisfactorily resolved your concerns. If you still have further queries or issues with our responses, please feel free to continue the discussion. We are more than happy to engage further.

Dear Reviewer RTBV,

Thank you again for your volunteer review. We believe our rebuttal has addressed your questions and concerns. As the deadline for the discussion phase is approaching, we would be grateful if you could let us know if there are any further questions, and we are happy to respond as soon as possible.

Dear Reviewer RTBV,

Regarding the newly raised questions, we've provided some experimental results and analysis, which we hope can address your concerns.

As the discussion period is about to end very soon, we'd be very grateful if you could take a moment to check our response and/or adjust your rating of our paper. Many thanks!

RTBV Q6: Insights of choose which layers to extend HATB.

As shown in the main peper and the response to the RTBV Q3, we can conclude:

1. Intergrating very few layers are not recommended because it leads to suboptimal performance due to the insufficient multimodal interaction. [Refer to the experiment results shown in Table 11 in Appendix.]
2. Intergrating a very large number of layers is also not recommended because it is less efficient and does not provide significant performance advantages. [Refer to the experiment results shown in Table 11 in Appendix.]
3. It is not recommended to choose a distribution that is too front-heavy or too back-heavy, as this can lead to poor performance. [Refer to the experiment results shown in the response to the RTBV Q3]
4. Choosing between 3 to 5 layers involves a tradeoff between performance and efficiency depending on the user's needs. integrating 4 layers in a uniform manner achieves a balance between efficiency and performance across various scenario. [Refer to the experiment results shown in the response to the RTBV Q3]

Thank you for your response. We provide explanations regarding your questions.

RTBV Q4: Video performance between mPLUG-Owl3 and Qwen2VL

Performance on LongVideoBench. As we showed in the response to RTBV Q1, we present the results on LongVideoBench, which is a pure long video benchmark. mPLUG-Owl3 achieves better performance than Qwen2VL and is significantly more efficient under various frame sampling strategies.

Performance on VideoMME. It is a hybrid benchmark, consisting of short, medium, and long videos. We take the result in the 768-sample frame setting from the technical report of Qwen2VL (only the overall score is reported), but we cannot reproduce the results locally due to the GPU memory usage far exceeds the capacity of a single GPU's memory without a specific design inference framework. In this comparison, the overall score of mPLUG-Owl3 with 128-frame sampling is slightly lower than that of Qwen2VL with 768-frame sampling.

To provide further insights, we present the result of mPLUG-Owl3 using 768-frame sampling. In conclusion, mPLUG-Owl3 achieves comparable performance across comprehensive video scenarios while maintaining a significant advantage in efficiency.

We also provide the detailed performance of mPLUG-Owl3 and Qwen2VL in 64-frame sampling in VideoMME to offer more insights. The results also suggest that mPLUG-Owl3 performs better than Qwen2VL in long video scenarios.

We again claim that Qwen2VL uses private training data, leading to an unfair comparison. As shown in the response to qTJ2 Q2, the performance comparison between mPLUG-Owl3 (trained under the exact same settings) and LLaVA-Next-OneVision suggest the advantage of mPLUG-Owl3 on various video benchmarks.

RTBV Q5: The source of Inference costs reduction

The memory savings of mPLUG-Owl3 during inference primarily come from two aspects.

1. Memory Bottleneck Thanks to the design of Hyper Attention, we avoid large memory allocation when applying attention to long visual sequences and avoid the main cause of out-of-memory issues.

In scaled dot-product attention, the memory allocation bottleneck occurs when the Query is multiplied by the Key.

For plain scaled dot product attention, visual and language content are concatenated.

$Q: (L_{img}+L_{txt})\times d$ $K: (L_{img}+L_{txt})\times d$

For hyper attention, cross-attention and self-attention are performed separately.

$Q_{txt}: (L_{txt})\times d$ $K_{txt}: (L_{img})\times d$ $K_{img}: (L_{img})\times d$

Therefore, Hyper Attention can reduce the memory bottleneck by

$\mathrm{min}(L^2_{img} + L^2_{txt} + L_{img}L_{txt}, L^2_{img} + 2L_{img}L_{txt})$

2. KV-Cache We reduce the memory allocation of the key-value cache during inference due to the sparse integration of HyperAttention.

A plain attention block stores KV-Cache, which costs:

$2\times N_{layer} \times (L_{img}+L_{txt}) \times d,$

during the whole inferce process.

In contrast, Hyper Attention layers are only integrated into four layers, thus we only need to store the full KV-Cache for those layers and the language cache for the other layers.

$2\times N_{layer} \times (L_{txt})\times d + 2\times 4 L_{img}\times d$

$N_{layers}$ denotes the number of layers in the language model.

$L_{img}$ and $L_{txt}$ denote the length of visual and language sequence.

$d$ denotes the hidden dimension of language model.

We first provide a detailed explanation of the frame sampling strategy:

• For NextQA and MVBench, which are short video benchmarks with video durations all less than one minute, we sample 8 frames and 16 frames as the other baselines [1,2,4].
• For datasets with longer videos, such as VideoMME, LongVideoBench, and MLVU, we explore the performance of mPLUG-Owl3 in handling long visual sequences by using higher frame rates and consistently sampling 128 frames. We also provide results for mPLUG-Owl3 using 16 or 32 frames, and it still outperforms the baseline models [1,2,3].

Given the total number of frames and video duration, we employ a uniform sampling strategy. Below is a table that annotates our method and the comparison methods with number of input frames.

In order to conduct a fair comparison in video scenarios and gain more insights, we first evaluate the performance of mPLUG-Owl3 on long video tasks with fixed 16 frames and fixed 32 frames. For ease of comparison, we present the highest results of the previous models, denoted by "best baselines". Since the methods of previous best baselines all use 16-frame sampling, we additionally evaluate the performance of the corresponding model with 32-frame sampling.

The experimental conclusions are twofold:

1. mPLUG-Owl3 still overcomes the best baselines under the 16-frame sampling setting, which suggests that mPLUG-Owl3 has advantages in video understanding scenarios.
2. The best baseline models do not achieve further performance improvements with 32-frame sampling. Together with the conclusions from Table 5 and Figure 4 in our manuscript, this suggests that Hyper Attention has better generalizability and robustness when modeling long sequences.

[1] Li, Kunchang, et al. "Mvbench: A comprehensive multi-modal video understanding benchmark." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Jiang, Dongfu, et al. "Mantis: Interleaved multi-image instruction tuning." arXiv preprint arXiv:2405.01483 (2024).

[3] Li, Feng, et al. "Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models." arXiv preprint arXiv:2407.07895 (2024).

[4] Jiang, Dongfu, et al. "Mantis: Interleaved multi-image instruction tuning." arXiv preprint arXiv:2405.01483 (2024).

We summarize the main factor contributing to mPLUG-Owl3’s superior capability in distractor resistance experiment:

1. Hyper-Attention allows self-attention to focus more on modeling language content, while cross-attention emphasizes the perception of visual content based on language semantics. Other methods mix these two within transformer blocks. As discussed in Section 4.4.1, these methods suffer from inter-image disruption caused by the causal attention between key images and irrelevant images. Table 5 in the manuscript also suggests that concatenating multimodal contents and modeling them with self-attention results in a poorer zero-shot performance in multi-image-text scenarios (NLVR2, Mantis-Eval).

2. The performance of a language model declines as position encoding is consumed. For long visual inputs, Hyper-Attention uses MI-Rope, which consumes less position indices and prevents the model's performance from sharply declining as the sequence grows. As shown in the response to 3S6H Q2, image-level rope has better performance on long visual content scenarios compared to patch-level rope.

Long visual sequences almost never cause position indices to exceed the model's limit. This is due to the following reasons:

1. Rope encoding is not learnable. Therefore, Qwen2 supports a 32k context input, and mPLUG-Owl3 can still supports a 32k context input. We only trimmed the training data to 4096 to improve training efficiency, without reducing the model's context window size.
2. Due to the design of MI-Rope, as claimed in the response to the 3S6H Q2, an image occupies only one media token placeholder's worth of indices in both text sequence and a visual sequence in mPLUG-Owl3, rather than the number of patches. with MI-Rope, mPLUG-Owl3 can support input with the number of images at the ten-thousand level.

As shown in Figure 1 (a), mPLUG-Owl3 is able to model up to 1000 image inputs on a single A100 GPU card, and far from exceeding the position indices. In conclusion, mPLUG-Owl3 can handle significantly longer image input than mainstream models. This also allows for a wider range of research topics to be explored based on mPLUG-Owl3.

The relative positions of patches within the image is preserved and effectively modeled by the model equipped with MI-Rope, because the use of MI-Rope decouples the position modeling into two modules:

• Intra-Image spatial information modeled by Vision Transformer. Since different patches of each image have already been assigned different positional encodings in the visual transformer, the relative positional information between patches is pre-encoded in the hidden states of the patches and can be perceived by the language model.
• Image-Text Interleaved information modeled by MI-Rope. MI-Rope assigns the same position index to different patches of the same image, which encourages the multimodal rotary embedding focus on representing sequential information of multimodal interleaved inputs.

Decoupling the two positional information allows the model to better model each.

We also follow the setting in Table 6, providing experimental results of mPLUG-Owl3, where each patch is assigned a specific position index (patch level rope) and all patches within an image share the same position embedding (image level rope). The results suggest:

1. Using image-level rope in single image tasks achieves comparable performance with patch-level rope without compromising the performance.
2. Image-level rope has greater advantages in video and multi-image tasks because the interleaved sequence information of images and text is better modeled.

Additionally, image-level ropes can save position indices and are beneficial for extending the model to receive longer visual content.

Dear Reviewer 3S6H,

Thank you for your volunteer review and insightful comments on the frame sampler setting, the working principle of MI-Rope, and the distractor resistance capability of mPLUG-Owl3.

For your questions and concerns, we have submitted point-by-point responses, and we trust that our responses have satisfactorily resolved them. If you still have further queries or issues with our responses, please feel free to continue the discussion. We are more than happy to engage further.

Dear Reviewer 3S6H,

Thank you again for volunteering to review our work. As the deadline for the discussion phase is approaching, we would be grateful if you could let us know if our responses have addressed your concerns and questions. Besides, if there are any further questions, we are happy to respond as soon as possible.

Dear Reviewer 3S6H,

Thanks again for volunteering to review our work. As the discussion phase is about to end very soon, we would be very grateful if you could take a moment to check our response and/or adjust your rating of our paper. Many thanks!