Rethinking Modality Alignment in Multi-Modal Large Language Models

1. The introduction of new components like the SA-Perceiver and reconstructive training may increase the model's complexity and computational requirements, which could be a barrier for adoption in resource-constrained environments.
2. The dual optimization objectives (reconstruction loss and autoregressive loss) might be challenging to balance, and the paper does not discuss how these are weighted or optimized together.
3. It's not fully addressed how VLSA scales with increasing model size or input complexity. There's a concern about whether the benefits would diminish as the model grows larger or the inputs become more complex.

1. Computational Complexity: Balancing the reconstruction and language modeling losses simultaneously can be challenging, potentially hindering optimization efficiency.
2. Limited Scalability: Given the computational complexity and the involvement of a two-stage alignment process, it is difficult to trust that this method has good scalability—an essential aspect for current VLMs.
3. Limited Experiments: The evaluation is restricted to conventional benchmarks (single-image input with textual output). Since the paper claims to improve visual-semantic understanding, could it demonstrate performance on benchmarks like BLINK [1] that involve visual prompts? Moreover, the authors assert that this is a general approach—have they considered testing on other interleaved benchmarks, such as DEMON [2] ?

[1] BLINK: Multimodal Large Language Models Can See but Not Perceive

[2] Fine-tuning multimodal llms to follow zero-shot demonstrative instructions

1. Several hypothesis lack theoretical/empirical justification: In the introduction and method section, the authors provide several hypothesis which aren't backed by theoretical justifications or empirical observations. Moreover, the introduction section is not very well written and it is hard to follow some of the claims. In the method section, while the section contains all technical details, the motivation behind some design choices is not very clear. For instance:

a) "the alignment between vision and text determines the lower bound of MLLMs’ performance" and "Therefore, we hypothesize that the alignment of LLMs’ cognition with visual semantics determines the upper bound of MLLMs’ performance". It is not clear why these two statements are true. There are several other factors which influence the performance of MLLMs and it is unclear why only the alignment between vision and text or the semantic alignment will be deciding the lower and upper bounds. It will be helpful if the authors can provide some additional context/ some sources where these claims are made or some experiments which help better support these claims.

b) "It’s important to note that this alignment focuses on mapping visual representations into the linguistic latent space, aiming for a distributional rather than a semantic alignment since vision inherently contains rich semantic information that is challenging to convey through text.": Could the authors please elaborate on what this means exactly? It will be good to have a more theoretical understanding of why the current alignment techniques are distributional rather than semantic. For example, looking at visualizations of the projections or looking at attention maps might help better support this claim.

2. Incomplete experiments and ablations: There are claims made in the paper which are not backed by experiments, and moreover a lot of design choices are not ablated well (it is unclear how the design choice was made). For instance:

a) The authors claim that the SA-Perceiver module leads to lower computational overhead during model inference, but no latency experiments/ theoretical analysis is provided as to why this is true. Given the fact that there is additional computation for computing the high resolution image features and the cross-attention with the low resolution image features, it is not clear why the architecture would be computationally more efficient during inference compared to a linear projector. It might be helpful if the authors can provide some latency analysis showing that inference using VLSA is faster compared to vanilla architectures. One easy way to show this could be to run inference over a fixed set of images and calculate the time taken per image for the different architectures and across different image resolutions.

b) The architecture of the SA-perceiver module lacks sufficient ablations as there aren't any ablations which compare using high resolution image features only, or other mechanisms or merging the high resolution features and low resolution features (for example concatenation followed by self-attention). Also having some analysis on how changing the lower resolution impacts performance would be interesting. Moreover, it is not clear why there needs to be a learnable parameter P in the low-resolution image features since that global embedding is not distinctly used anywhere else in the architecture. Adding an ablation with only high-resolution image features, high-resolution features + self-attention, and [high-resolution features,low-resolution features] + self-attention could be a useless analysis to conduct.

c) In the cognition alignment module, there are no ablations which compare using only the codebook task vs using the rgb task. Also, it would be helpful to have some more analysis on how exactly these tasks help other than just looking at the performance boosts on benchmarks. For example, having more results which show that training with codebook data is able to improve semantic understanding of the image, or showing that training with rgb value prediction leads to improved low-level image recognition might add more support to the architecture design choice.

d) All experiments and ablations are only done with the Llava-Next architecture and only LLama3-8B as the LLM, so it is unclear if these improvements in benchmarks also generalize to other MLLM architectures which can support the SA-Perceiver module. It would be good to have more experiments across different LLM architectures and sizes (Vicuna 7B, LLama 3B, etc) and also across different architectures (Shikra, Qwen2VL, Mini-GPT4) etc.

3. Architecture and training paradigm is quite complex: There are several additional components in the architecture and it uses a 3 phase training procedure, adding significant overhead in terms of number of additional modules needed (LDM, VQ-VAE etc) and also training time. While the performance gains on benchmarks is impressive, it also severely limits the generalizability and extensibility of this architecture. It might be interesting to look at the trade-off between the complexity and performance by simplifying the architecture a bit.

4. Minor nitpicks: In equation 5, the shape of of TargetVQ should be (sxh'xw') assuming the X operation refers to dot product.

1. The designs of some key components in the method are not clearly motivated and lack novelty. In particular, the two-scale image representation seems to be widely used in vision encoder design. Moreover, the semantic alignment for MLLMs is not new. Several recent works have added new finetuning tasks that aim for the object or visual relation prediction to improve the cognition capability, e.g., [a] Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning, CVPR 2024, and others.
2. The proposed method employs several pre-trained models (Stable Diffusion, VQ-VAE) and a complex architecture/training pipeline. Such a design choice lacks justification or ablation analysis. For example, it is unclear why the so-called SA-Perceiver needs to take such a structure (also see comments on ablation below). In addition, while the reconstruction loss makes sense, it is also unclear why a latent diffusion model is necessary for this loss term. Furthermore, what is the advantage of choosing VQ-VAE code indices compared to predicting objects in the finetune tasks?
3. The experimental evaluation seems lacking in the following aspects:
   • Regarding the experimental setup, the paper only considered the integration between LLaVA-Next and the proposed VLA strategy. As this method is an add-on for MLLMs, it should also report evaluations on other MLLMs to show its generality.
   • The comparisons with previous methods (except LLaVA-Next) seem unfair, as this method has access to several additional pre-trained models during training, a much better LLM (LLaMA3), and additional OCR training data. The setting alignment with LLaVA-Next seems also questionable (Line 396).
   • Table 3 shows mixed results from the ablation analysis. As the author observed, the proposed vision encoder and reconstructive training do not always work as expected.
   • The ablation study should also include comparisons with common baseline choices for the module design, including the vision encoder architecture, reconstruction process, and fine-tuning tasks.