Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models

We appreciate your valuable comments! In the following rebuttals, we address all the comments you pointed out. We will definitely include these clarifications in our manuscript to enhance understanding in the potential camera-ready version.

Q1. The introduction of Meteor-Mamba increases the model size, which contradicts the abstract. Additionally, Meteor's inference cost is higher than models like LLAVA and InternLM-XComposer2.

A1. We believe the Meteor-Mamba 130M size is a marginal increase compared with the large language model size of 7B. Nonetheless, we will modify the expression to "without 'significantly' increasing model size" to clarify this point.

Regarding inference time, please refer to Answer 1 of "Reviewer Hj2J," which describes the comparison of inference times. Based on this table, we can say that Meteor's inference cost is not higher than commonly used LLVMs.

Q2. In Figure 3, what is the Meteor-Multimodal Language Model in the Second Training Step? Was it initialized with pre-trained MLLMs weights?

A2. Yes, the pretrained language model is used in the second training step. The term "Meteor-Multimodal Language Model" in Figure 3 indicates that as training progresses, the pretrained language model evolves into a multimodal language model. We will add an explanation to clarify this in the manuscript.

Q3. How does the Mamba architecture benefit Meteor? The experiments do not demonstrate the advantages of the Mamba structure.

A3. For detailed reasoning capabilities of Meteor, please refer to Answer 2 tables of "Reviewer XZUR," which describes Meteor's complex reasoning capabilities on challenging benchmarks such as MM-Vet and LLaVA-W. In these tables, we showed the performance differences between with or without Meteor-Mamba for complex reasoning capabilities.

Q4. I am puzzled by the performance achieved with only 2.1M instruction tuning. Please provide a detailed description of the training process, the data used, and the weights of each model employed.

A4. Below are the details of the training process, datasets used, and models employed in each training step:

1. Training Process:

In the first training step, we only train Meteor-Mamba, the visual projector, and the tor projector. We autoregressively train only the partial segments of multifaceted rationales between special tokens $<tor>$. In the second training step, we train all parameters except the vision encoder. Only tor tokens are propagated from Meteor-Mamba into Meteor-MLM without explicit rationale. We autoregressively train the answer parts for the given question prompts.

2. Used Dataset

We used a curated dataset of 1,059,382 (1.1M) Q-R-A triplet samples.

The dataset breakdown is as follows:

--------------------------------------------
Real-World Image: 338K
Document & Chart & Diagram & Sign & Symbol: 379K
Math: 342K
     Math with Vision: 165K
     Math with Text only: 177K
--------------------------------------------

- ShareGPT4V-Caption (72507, 73K)
- ShareGPT4V-Instruction (266072, 266K)
- MiniGemini-Instruction (26885, 27K)
- DocDownstream (298748, 299K)
- DocReason (53065, 53K)
- GLLaVA (162378, 162K)
- MathVision (2992, 3K)
- MathInstruct (81496, 81K)
- MathPlus (95239, 95K)

3. Model employed in Each Training Step

• Vision Encoder: CLIP-L/14
• Pretrained Large Language Model: InternLM2-7B

In the first training step, we freeze the vision encoder and the pretrained large language model. In the second training step, we freeze only the vision encoder.

We hope this rebuttal can improve the "Rating" score from Negatvie to Positive.

We appreciate your valuable comments! In the following rebuttals, we clarify the points you raised. We will incorporate this content into our manuscript in the next potential camera-ready stage.

Q1. How about the inference time against common MLLMs?

A1. We evaluated the inference time under equal resource environments: Intel(R) Xeon(R) Gold 6230, RAM 512GB, and NVIDIA RTX A6000, with flash attention applied. Despite Meteor using additional Meteor-Mamba module, it did not slow down the inference speed. This is because once Meteor-Mamba's embedded rationale features are acquired, no additional propagation of Meteor-Mamba is needed. Only the autoregressive decoding process in Meteor-MLM is required.

Q2. Does this structure can be scaled up? Will Meteor be better when combining high-resolution image encoders, e.g., LLaVA-HR

A2. The table below shows the results of scaling up the model size of Meteor-Mamba.

We also applied Meteor to LLaVA-HR to adapt to high image resolution.

Aligned with the results in MM1 [R1], increasing image resolution significantly improves vision-language performance. From these two tables, it is evident that enhancing image resolution with Meteor-Mamba is more effective than merely enlarging the Meteor-Mamba architecture size.

Q3. To be honest, the writing of the paper should be improved. I take a lot of time to understand the technical details. The method should be better organized.

A3. We appreciate your feedback regarding the organization of the technical details. In the revised version of our manuscript, we will reorganize the technical details and the methods section to improve clarity and understandability.

References

[R1] McKinzie, Brandon, et al. "Mm1: Methods, analysis & insights from multimodal llm pre-training." arXiv preprint arXiv:2403.09611 (2024).

We hope this rebuttal can improve the "Rating" score more positively.

Thanks for authors'rebuttal. The provided results further address my concerns. The newly added comparisons should be included in the final version. Therefore, I would like to improve my score to accpet.

Thank you for your insightful comments! We will incorporate the following rebuttal into our manuscript to enhance overall understanding in the potential camera-ready version.

Q1. Writing: All Figures (fonts, line widths, etc.) in the paper are not uniform, there is too much white space. And even Figure 1 shows strange tables and font proportions (fonts are also not clear).

A1. We will ensure uniform font formats and line widths for all figures and remove unnecessary white space as you suggested. Additionally, we will correct the tables and font proportions in Figure 1 to improve clarity.

Q2. Motivation: The paper employ the Mamba architecture to embed lengthy rationales containing abundant information, but without setting and demonstrating task scenarios that existing LVLMs does not achieve that require extreme long sequence modeling capabilities.

A2. We would like to clarify that our focus is not on generating long rationale text, but on embedding multifaceted information used in larger models with a variety of forms. This includes fundamental image understanding, real-world knowledge about common sense and non-object concepts, and step-by-step procedures for solving complex questions.

For detailed reasoning capabilities of Meteor, please refer to Answer 2 of "Reviewer XZUR," which describes Meteor's complex reasoning capabilities on challenging benchmarks such as MM-Vet and LLaVA-W.

Q3. It appears to share the same motivation as some existing methods, for instance, Vary-toy. Please provide a more detailed explanation of how the proposed "traversal of rationale" differs from the design implemented in Vary-toy.

A3. Vary-toy generates a new vision vocabulary network trained on a smaller language model, which is later merged with CLIP to train a larger language model. Consequently, Vary-toy aims to achieve more fine-grained vision perception through an expanded vision tokenizer.

In contrast, our design of traversal of rationale is motivated by the goal of embedding multifaceted rationale for not only vision perception but also common sense, non-object concepts, and step-by-step procedures. Specifically, we employ the Mamba architecture to embed long sequential rationales into $<tor>$ tokens, which are subsequently propagated into Meteor-MLM. These tokens plays a role in conveying embeded multifaceted rationale into Meteor-MLM which ultimately answers the question.

Q4. Method: Show more comparison with the MMLM models based on Mamba, like Cobra or VL-Mamba.

A4. The table below compares our model to Cobra, VL-Mamba, RoboMamba [R1], ML-Mamba [R2]. We will include the results of these Mamba-based LLVMs in the updated manuscript.

References

[R1] Liu, Jiaming, et al. "RoboMamba: Multimodal State Space Model for Efficient Robot Reasoning and Manipulation." arXiv preprint arXiv:2406.04339 (2024).

[R2] Huang, Wenjun, and Jianguo Hu. "ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2." arXiv preprint arXiv:2407.19832 (2024).

We hope this rebuttal can improve the "Rating" score more positively.

Thank you for your valuable comments. We will incorporate the following rebuttal into our manuscript to enhance overall understanding in the potential camera-ready version.

Q1. Figure 3 (Stage-One Pre-Training): More clarity and intuition behind the model design are needed. Given that the training is formulated as an autoregressive token generation task: a) Are TOR embeddings predicted as special tokens [TOR]? b) Or are losses on these tokens bypassed? If the latter, a potential drawback is the inability to assess the model's performance after the first stage of training, as it cannot produce complete sentences/rationales.

A1. Among options a) and b), the correct approach is b). This means that special tokens $<tor>$ are not predicted tokens in the autoregressive loss but are instead input tokens in Meteor-Mamba. These tokens play a crucial role in embedding the multifaceted rationale interleaved between the special tokens $<tor>$. In other words, the model learns to predict the partial segments of a multifaceted rationale that are between the $<tor>$ tokens in the autoregressive loss.

As you pointed out, there may be a potential drawback in "Meteor-Mamba" due to its inability to produce full rationale sentences. However, we argue that "Meteor-MLM" is ultimately used to generate text responses, so the text generation quality of "Meteor-MLM" reflects on the reasoning capability learned from "Meteor-Mamba." We have provided detailed validation results in Answer 2 of Question 2, as below.

Q2. Benchmarks and Reasoning Capabilities: The proposed benchmarks may not sufficiently test complex rationale reasoning (e.g., current tasks are mostly recognition or OCR-heavy). Recommendation: Conduct further studies in the final version on reasoning-oriented benchmarks such as Visual Commonsense Reasoning (VCR)

A2. We respectfully disagree with the assertion that "the proposed benchmarks may not sufficiently test complex rationale reasoning." The evaluation benchmarks MM-Vet and LLaVA-W, as presented in Tables 1-3, already include challenging measures of complex reasoning based on GPT-4 evaluation.

For MM-Vet, the sub-benchmark "Language Generation" in Table 2(d) specifically represents the quality of text generation required to answer complex questions. Additionally, LLaVA-W contains a sub-benchmark explicitly titled "Complex Reasoning." Below, we present a table showing the sub-benchmarks of MM-Vet and LLaVA-W compared with Meteor and the previous SOTA LLVMs within equal size (Meteor uses about 2 times more dataset samples than them. In addition, they are not performant well to these challenging benchmarks.). This table also includes an ablation study where "Meteor-Mamba" is removed, highlighting the effectiveness of "Meteor-Mamba" in enhancing reasoning capabilities.

We believe the reasoning questions in MM-Vet and LLaVA-W are significantly more complex than those in the Visual Commonsense Reasoning (VCR) benchmark you recommended. Recent models, such as OV-Grounding and GPT4RoI, have nearly matched human performance on the VCR leaderboard, indicating the need for more challenging questions, such as those found in MM-Vet and LLaVA-W. These benchmarks remain unsaturated and continue to pose difficult problems evaluated using GPT-4, ensuring a rigorous assessment of reasoning capabilities.

Q3. Ablation Studies (Table 3a): Clarification needed: Did the baselines undergo the same first-stage pre-training as Mamba? If so, what were the training objectives for each method?

A3. Yes, all baselines underwent the same first-stage pre-training as Mamba. The training objectives for each method were identical: "The partial segments of a multifaceted rationale between special tokens $<tor>$ are learned to predict in autoregressive loss, where the tokens $<tor>$ are not predicted tokens in autoregressive loss."

Q4. Comparison Methodology (Figure 2): Line 147 indicates that the training data includes annotations from ChartQA and AI2D. Some results (e.g., Gemini-Pro and GPT-4V) are zero-shot and not directly comparable. Recommendation: Revise the plots to accurately reflect these differences in training/evaluation conditions.

A4. For Gemini-Pro and GPT-4V, the training datasets are unknown as they are totally closed-source LLVMs, making it uncertain whether their results are truly zero-shot. However, for Table 1, we will explicitly mention that the ChartQA and AI2D performances for LLaVA family models are evaluated under zero-shot conditions. This clarification will help accurately reflect the differences in training and evaluation conditions.