Unleashing the Power of Selective State Space Models in Vision-Language Models

1. The novelty is limited with the following reasons: (1) The model is based on Cobra, with minor changes on the concatenation of visual features and projector. (2) The scan directions are from VMamba, only the stitch-scan is novel.

2. Although the sequence-level concatenation improves the performance, it poses a great concern on the efficiency of the model, but the authors did not provide the inference speed, computational cost, and memory cost comparisons. Though the Mamba has linear computational complexity, longer sequence indeed increases the FLOPs and memory consuption, and the heavy projector also introduces additional costs. As a result, directly compare the model with existing methods such as Cobra without comparing the efficiency is unfair.

3. In Figure 1, directly comparing LLaVA-1.5 with MambaVLM to demonstrate the effective of Mamba and the superiority on training time is unfair, as MambaVLM uses better DINOv2-SigLIP encoder.

4. In lines 215~235, "regardless of how many channels ... loss of visual information" is overstated, lacking precise theoretical evidence to support the claims. Bottleneck-structures are widely used in networks such as ResNet, and according to information bottleneck principle, it is no clear evidence to state that the compression of channels will definitely lose the valuable information. Please reword.

5. In Table 1, some results (62.6, 76.3) of MambaVLM is not the best and should not be bolded. Please correct them.

1. The performance comparison with the original LLaVA is somewhat unfair, as the method in the paper uses two visual encoders. It would be better if a version with only ViT-CLIP could be provided.
2. The method description in the paper is unclear; perhaps I missed where it explains how Mamba-VLM + Vicuna is implemented. It seems that if Vicuna is used, only the Mamba projector is related to Mamba. Of course, I also understand that the performance of VLMs is highly dependent on the performance of the LLM, and Mamba as an LLM is still relatively weak.

1): Limited Novelty in Technical Contribution: While the paper proposes a "visual long sequence" as part of the framework, a significant body of literature already exists on augmenting visual features using ensembles of different visual encoders, as demonstrated in works such as [A-D]. The design of the Mamba projector, specifically its cross-stitch scanning scheme that concatenates four scanning paths, seems heuristic rather than theoretically grounded.

2): Unclear Motivation for Mamba Projector: The Mamba projector, the primary technical contribution of this paper, has an unclear motivation. The 1x1 convolutional MLP layer can be treated as a full attention layer, suggesting that the Mamba projector is an approximation. Lines 250–253 argue that "a simple MLP layer may not be able to accomplish sufficient vision-language alignment and interaction of different visual features. Therefore, we devise a lightweight mamba projector…" However, this rationale does not sufficiently justify the addition of the Mamba projector.

3): Unfair Experimental Comparisons: For instance, in Table 4, using a longer visual sequence generally increases latency. Models such as TinyLLaVA and MobileVLMv2 should be substituted with the Mamba LLM. In Table 2, MambaVLM shows superior performance, largely attributed to encoder ensembling—a common approach in the literature.

4): Presentation Quality: The paper’s overall clarity and presentation could benefit from further refinement.

References:

[A]: BRAVE: Broadening the Visual Encoding of Vision-Language Models, ArXiv.

[B]: Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, CVPR 2024.

[C]: Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders, ArXiv.

[D]: Law of Vision Representation in MLLMs, ArXiv.

• The contributions are vague; it would be better to clearly summarize the contributions of this paper at the end of the Introduction. This article simply replaces the traditional MLLM with the Mamba model, and the proposed Stitch-Scan is merely a data augmentation stitching method.

• The experiments are insufficient. The core argument of this article is: "we first construct visual long sequences with multiple vision encoders, which not only enrich visual representations but also leverage the advantages of Mamba in handling long sequences. Notably, this design will not undermine the efficiency obviously, which is in stark contrast with the common cognition of Transformer-based MLLMs." Is there any experimental or theoretical support for this conclusion? How much is "not undermining the efficiency obviously" specifically? It is recommended that a row be added to Table 4 so that the visual tokens of MambaVLM and MobileLLaMA-2.7B are also consistent at 144, which would support the above point.

• Formula 7 is expressed non-standardly; do not mix mathematical symbols with code.

• In Formula 8, Hv = Merge(Hv1, Hv2, Hv3, Hv4), the Merge method is not explained in the text. What specific merging technique is used, just a simple concatenation?

• In Table 1, the Qwen-VL model outperforms MambaVLM in performance on TextVQA and VQAv2 with a data scale of 665K. Typically in papers, bold numbers indicate the best results obtained by models, but this is not the case in your table. If the bold numbers have a special meaning, please explain this in the text. Additionally, the same issue occurs in Table 2.

1. The proposed method is likely to be dependent on vision encoders. It would be more solid if the author could conduct additional experiments on encoders other than DINOv2 + SigLIP. Also, the author does not show how proposed method perform on single vision encoder MLLM.

2. There is not enough ablations experiments on the scanning orders. For example, no comparison with only using Hv1.