MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model

We would like to express our sincere gratitude for your insightful comments and the recognition of our work. We have carefully considered your suggestion and provide a detailed response below.

1. it would be recommended to conduct experiments comparing the performance of these different discrete representation techniques, such as VQGAN or VQVAE.

Thank you very much for your insightful suggestion. We fully acknowledge the importance of validating the efficacy of the discrete visual symbol sequences utilized in MaVEn by comparing them with other discrete representation techniques, such as VQGAN and VQVAE. These methods are indeed prominent in the field and have demonstrated significant success in various applications.

In response to your recommendation, we have conducted additional experiments to compare the performance of MaVEn using different discrete representation techniques, including VQGAN and VQVAE. Additionally, we explored the potential of combining these techniques to further expand the vocabulary of the large language model (LLM).

Below are the results of our comparative experiments:

Our conclusions are as follows:

1. Using SEED as the discrete visual token yields better performance compared to VQGAN and VQVAE.
2. Combining different discrete tokenizers can enhance the model's performance. We believe this improvement is due to the different visual semantic information encoded by the distinct codebooks. By integrating multiple codebooks, we achieve a richer and more comprehensive visual semantic representation, which in turn helps improve the model's overall performance.

We appreciate your valuable feedback, which has significantly contributed to enhancing the robustness of our findings. We look forward to any further comments or suggestions you may have.

Thank you for your thoughtful consideration.

1. Would the adjustments to the Visual Projector affect the performance of the Patch Selector?

The adjustments made to the Visual Projector in Stage 3 will not affect the performance of the Patch Selector. As shown in Figure 2(b) in our paper, the process is designed such that we first use the Patch Selector to select the important patch tokens. Only after this selection has been made, we then input the selected patch tokens into the Visual Projector. This sequential process ensures that the modifications made to the Visual Projector do not influence the operation of the Patch Selector.

2. Directly selecting patches based on the similarity between the patch and the discrete token may be a simpler and more effective approach.

Thank you for your insightful suggestion.

1. We have indeed experimented with a similar approach where we directly select patches based on the attention weights between the <EOS> token in the discrete visual token sequence and all other continuous visual tokens. However, the results are shown below which were not satisfactory. This indicates that directly using attention weights may not be an effective method to measure the similarity between discrete tokens and continuous tokens.

2. On the other hand, Grounded SAM is designed to locate relevant visual regions based on textual semantics, and given the fact (as shwon in the figure 3 in the rebuttal pdf) that for the paired image-text sample, the semantics encoded by the text captions is similar to semantics encoded by the visual discrete tokens, we decided to use Grounded SAM to generate pseudo-labels. This allows the patch selector to learn the similarity between visual discrete tokens and patches more effectively.

Overall, we appreciate your suggestion and believe that the use of Grounded SAM to train the patch selector, although seemingly redundant, is a necessary step to ensure the effectiveness of our model.

3. The role of the continuous tokens has not been well validated.

Your comments are greatly appreciated, and we will address your concerns in two parts:

1. Clarification of Figure 6 and the importance of discrete Tokens: We apologize for any confusion caused by Figure 6. In fact, the text question in the case presented in Figure 6 is "What is the similarity between Figure 1 and Figure 2?" This case was to highlight the challenges faced by recent MLLMs, particularly in multi-image contexts, where the semantic granularity of the continuous visual representations and text tokens differ significantly. In this specific case, the American flag represents a higher-dimensional, coarse-grained semantic entity that continuous visual tokens struggle to effectively encode alone. This results in the MLLMs paying less attention to continuous visual tokens during the decoding phase. However, upon introducing discrete visual tokens, which align more closely in semantic granularity with text tokens, the MLLM was able to better grasp the high-level semantic from the discrete tokens, thereby focusing more attention on these tokens and enhancing its ability to establish semantic associations across multiple images.

   • To further verify the semantics of the discrete visual tokens that the MLLM focuses on, as illustrated in the figure 3 within the rebuttal PDF, we collected images that contain the discrete visual tokens targeted by the LLM. We discovered that these images consistently feature the American flag.

2. The Role of Continuous Tokens-Encoding Fine-Grained Visual Details: We are not suggesting that continuous visual tokens are useless. In fact, continuous visual tokens and discrete tokens are complementary and indispensable. Continuous tokens encode a amount of fine-grained semantic information in the image, so when the model faces scenarios that require understanding of fine-grained details in the image (where discrete visual tokens often do not contain this information), we actually need continuous visual tokens. To better validate this point, we conducted the following experiment:

   • We randomly collected 500 images from the COCO dataset and tasked a GPT-4 model to generate questions about detailed object information in each image (e.g., asking about the shape, color, number, and size of certain objects) along with four different options and the correct answer. We then tested the performance on this dataset using MaVEN with only discrete visual tokens, only continuous visual tokens, and both, as shown below table, we found that the performance of the model using only discrete visual tokens was very poor, while the performance of the model using only continuous tokens was very close to that of the model using both, which also demonstrates the importance of continuous visual tokens for understanding detailed image information.

   • As shown in Fugure 1 in the submitted pdf in this rubuttal, we have also visualized the attention distribution of MaVEn for both discrete and continuous visual tokens when the model is asked about some details of the image (e.g.,"what is the color the dog in this image?"). We found that MLLMs also pay attention to continuous visual tokens. Which also suggests that the continuous visual tokens provide the fine-grained detial informations for our model.

4. In line 202, '567' should be '576'.

We appreciate your attention to detail and have corrected this error in the revised version of the paper.

Dear Reviewer TdkD,

First and foremost, please allow me to extend our deepest appreciation for the time and effort you have devoted to reviewing our paper.

Moving forward, as the discussion phase is approaching its end, we are confident that we have comprehensively addressed the concerns you raised. We would greatly appreciate it if you could take a moment to review our responses. Your insights are important to us, and we are eager to hear your thoughts on the revisions we have made.

Thank you once again for your attention and assistance.

Thank you for your recognition of our work and for your insightful questions.

1. Why not directly operate on continuous features to reduce feature redundancy ?

This is an excellent question. Directly reducing redundancy in continuous features can be approached in two main ways:

1. Token Selection with global semantic: This could be done by using the attention weights from the image global semantic token (e.g., <EOS> token) to select important continuous visual tokens [1], [2].
2. Merging Visual Tokens: This can be done using latent queries (e.g., InstructBLIP[3] used q-former to extract the fixed length visual continuous sequence) or convolutional network (e.g., QwenVL[4]) to merge long sequences of continuous visual tokens.

We have experimented with these methods as per your suggestion, and the results are as follows:

We found that these methods were less effective than our proposed approach, which combines discrete visual tokens with continuous tokens. From these results, we have the following inferences:

1. Compared with the first type of methods, our method is similar to them but is more efficient and accurate because our method utilizes coarse-grained discrete visual features and fine-grained continuous features to encode complementary information. In our method, discrete visual features capture high-level, coarse-grained information (e.g., "snowman," "American flag" as shown in Figure 6 in our paper), while continuous features capture fine-grained details of the image. In contrast, traditional methods based on global image semantic representations for token selection lack high-dimensional semantic guidance, resulting in lower accuracy in token selection.

2. The second type of method, which merges visual information, compresses the data and loses important information, thereby impairing the MLLM's understanding of the image.

To further illustrate the superiority of our method over traditional token selection methods, we conducted an experiment. We randomly selected 500 images from the COCO dataset and used Grounding SAM to segment relevant regions based on textual semantics of image caption as ground truth. We then measured the accuracy of selecting 20%, 40%, 60%, and 80% of tokens that hit the ground truth region for both approaches. As shown in below table, our method significantly outperformed the global visual semantic-based token selection method, demonstrating its effectiveness.

2. It's not quite clear how discrete tokens help non-single image understanding. The obvious advantage of using discrete tokens is to improve efficiency only?

Thank you for your insightful question.

1. To better address your concern, we would like to draw your attention to Figure 6 of our paper, which visualizes the Average Attention Weights with Only Continuous Visual Tokens and the Average Attention Weights with Multi-granularity Hybrid Visual Encoding. The text question in the case presented in Figure 6 is, "What is the similarity between Figure 1 and Figure 2?" This case highlights the challenges faced by recent MLLMs, particularly in multi-image contexts, where the semantic granularity of continuous visual representations and text tokens differ significantly, making it difficult for MLLMs to understand and capture high-dimensional semantic information from images. In this specific case, the American flag represents a higher-dimensional, coarse-grained semantic entity that continuous visual tokens alone struggle to effectively encode. As shown in the visualization of Average Attention Weights with Only Continuous Visual Tokens in Figure 6, this results in the MLLMs paying less attention to continuous visual tokens during the decoding phase.

2. Upon introducing discrete visual tokens, which align more closely in semantic granularity with text tokens, the MLLM was able to better grasp the high-level semantics from the discrete tokens. This alignment allowed the model to focus more attention on these tokens, thereby enhancing its ability to establish semantic associations across multiple images.

3. Moreover, to further verify the semantics of the discrete visual tokens that the MLLM focuses on, as illustrated in the figure within the rebuttal PDF, we collected images that contain the discrete visual tokens targeted by the LLM. We discovered that these images consistently feature the American flag. Therefore, it can be inferred that the semantics of the corresponding discrete visual tokens are associated with the American flag.

In summary, while discrete tokens do improve efficiency, their primary advantage lies in their ability to bridge the semantic gap between multi-image visual representations and text representations. This alignment enhances the model's ability to understand high-level semantics and establish meaningful associations across multiple images, ultimately improving multi-image understanding.

Reference

[1] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models.

[2] Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations.

[3] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

[4] Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities*

Dear Reviewer od3A,

Firstly, we would like to extend our sincere gratitude for the time and effort you have dedicated to reviewing our manuscript. Your insights and feedback are invaluable to us. As the discussion phase is nearing its conclusion, we believe we have addressed the concerns you raised in your review. We would be grateful if you could review our latest responses at your earliest convenience. Your further comments would be immensely helpful in refining our paper and moving forward in the review process.

Thank you once again for your attention and assistance.

1. Not compare some of the latest methods

Thank you for your valuable comments. We have reproduced the results of mini-Gemini, MiniCPM-V, and InternVL and compared MaVEn with them. It is important to emphasize that our experiments were based on LLaVA 1.5. To validate our method's effectiveness on the most current MLLMs, we further retrained MaVEn on the latest SOTA single-image MLLM LLaVA Next, we named it MaVEn-NEXT. The results are as follows:

We observed that MaVEn remains superior multi-image performance which demonstrating the effectiveness of our method. Moreover, MaVEn-NEXT shows improved single-image performance, reaching SOTA or comparable results.

2. Testing on additional multi-image benchmarks

Thank you for your suggestion. Indeed, MMBench-Video and MME-Video were released after the NIPS submission deadline. Moreover, based on your suggestion, we evaluated our model on MMBench-Video and MME-Video during the rebuttal period. During the evaluation, we extracted 8 frames from each video as input:

We found that MaVEn performs better than MLLM models such as mPLUG-owl2, Qwen-VL, Idefics2-8B, and its performance is comparable to that of InternVL-Chat-1.5.

For the MME-Video results, please refer to Table 1 in the rebuttal pdf, where MaVEn also achieve competitive performance.

3. Figure 6 is unclear.

1. Clarification of Figure 6: We apologize for any confusion caused by Figure 6. In fact, the text question in the case is "What is the similarity between Figure 1 and Figure 2?" . In this case, the American flag represents a coarse-grained semantic entity that continuous visual tokens struggle to effectively encode alone. This results in the MLLMs paying less attention to continuous visual tokens during the decoding phase.
2. Importance of continuous visual tokens: When the model faces scenarios that require understanding of fine-grained details in the image, MLLMs actually need continuous visual tokens. To better validate this point, as shown in Fugure 1 in the submitted pdf in this rubuttal, we have visualized the attention weights distribution of MaVEn when the model is asked about the details of the image. We found that MLLMs also pay attention to continuous visual tokens.
3. Moreover, we conducted the following experiment: We randomly collected 500 images from the COCO dataset and tasked GPT4-o to generate questions about detailed object information in each image (e.g., asking about the shape, color, number, and size of an objects) . We then tested the accuracy on this dataset using MaVEN with only discrete visual tokens, only continuous visual tokens, and both, as shown in below table. We found the performance of the model using only discrete visual tokens was very poor.

4. Report the computational complexity compared to other methods.

In response, we have conducted a thorough evaluation:

1. Experiment Setting: Specifically, we set the input image size to 336x336 and the number of text input tokens to 24. We then measured the throughput and FLOPs of 10 inference step for different numbers of images (ranging from 2 to 8) across several models: LLaVA 1.5, MaVEn, QwenVL, InternVL. The inference batch size is 1 and was performed on single 80G A100 GPU.
2. Experiment Results: As shown in Figure 2 of the submitted pdf file, we observed that as the number of images increases, MaVEn exhibits higher efficiency. This is primarily because MaVEn encodes a lower number of continuous visual tokens, which reduces the computational burden.

5.Directly use Grounding Sam for token selection?

Very insightful question!

1. First, we want to highlight that Grounded SAM is engineered to segment image regions based on their corresponding textual information. However, if user instructions fail to provide explicit textual semantic cues for image segmentation (such as "What are the differences between image one and image two?"), it may lead to inaccurate region segmentation by Grounded SAM.
2. To fully address your concern, we tested the direct use of Grounded SAM for selecting image patches in both multi-image and single-image benchmarks. The results are as follows:

The results indicate a significant decline when using Grounded SAM for selecting image patches, highlighting the effectiveness of patch selector.

6. training cost?

We used 8*80G A100 GPU for training, which overall took about 122 hours. We give more detials in the table2 of the rebuttal pdf.

Dear Reviewer 9o1r,

We would like to sincerely thank you for the time and effort you have dedicated to reviewing our paper. Your insights and feedback have been invaluable in helping us improve our work.

As the discussion phase is nearing its conclusion, we believe we have addressed your concerns in our recent replies and would greatly appreciate it if you could take a moment to review our responses. Your insights are important to us, and we are eager to hear your thoughts on the revisions we have made. Thank you once again for your time and consideration.

Dear Reviewers,

We would like to kindly remind you to review our response to your comments and concerns. We have made our best efforts to address all the points raised by the reviewers and have provided detailed explanations and additional experiments to support our work. The valuable insights and suggestions provided by the reviewers have helped us to strengthen our contributions and clarify the key aspects of our research. We are eager to engage in further discussion with the reviewers and welcome any additional feedback or questions you may have. Your expertise and comments are highly appreciated.

Thank you once again for your time and consideration.

1. The system can be a bit over complex for serving & compare computation complexity with other existing models.

Thank you for your insightful comments. We appreciate your concern and would like to address it comprehensively. We acknowledge that the training process of our model might appear complex. However, we have conducted a thorough analysis comparing our model to recent SOTA models including QwenVL[1], InternVL[2], LLaVA1.5[3] in terms of training data size, training GPU count, inference FLOPs and latency . To evaluate the FLOPs and latency performance, we set the input image size of 336x336 and 24 text input tokens. We measured throughput and FLOPs over 10 inference steps using 4 input images, with an inference batch size of 1, on a single 80G A100 GPU.

Table 1. Comparision with different MLLMs on training data size, training GPU count, FLOPs, Latency and benchmark performance.

As shown in below table 2 and table 3. We also report trends in FLOPs and latency across different numbers (from 2 to 8) of image inputs.

Table 2. FLOPs across different numbers of image inputs for various models.

Table 3. Latency across different numbers of image inputs for various models.

We have following conclusion:

1. The results in Table 1 demonstrate that our approach actually incurs lower overhead in both training and inference stages compared to these models. This suggests that, contrary to initial impressions, our method is indeed practical and efficient for deployment and maintenance in real-world scenarios.
2. The results in Tables 2 & 3, we observed that as the number of images increases, MaVEn exhibits higher efficiency. This is primarily because MaVEn encodes a lower number of continuous visual tokens, which reduces the computational burden.

We hope this detailed analysis alleviates your concerns about the complexity of our model in practical applications.

2. How does MaVEn compare with other multi-granularity approaches?

In response to your recommendation, we have conducted additional experiments to compare the performance of MaVEn using different multi-granularity techniques, where we utilize VQGAN and VQVAE to replace the SEED tokenizer. Additionally, we explored the potential of combining these techniques. Below are the results of our comparative experiments:

Our conclusions are as follows:

1. Using SEED as the discrete visual token yields better performance compared to VQGAN and VQVAE.
2. Combining different discrete tokenizers can enhance the model's performance. We believe this improvement is due to the different visual semantic information encoded by the distinct codebooks. By integrating multiple codebooks, we achieve a richer and more comprehensive visual semantic representation, which in turn helps improve the model's overall performance.

[1] Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities.

[2] InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks.

[3] Improved Baselines with Visual Instruction Tuning.

Dear Reviewer Rxew,

Thank you very much for taking the time and effort to review our paper. We sincerely appreciate your valuable feedback and insights.

We wanted to inform you that we have provided our responses in the comments section. We would greatly appreciate it if you could take a moment to review our replies.

Thank you once again for your time and consideration.

Thanks for the rebuttal! The comparison looks nice, I changed the rating to 7 (Accept)