EMMA: Efficient Visual Alignment in Multi-Modal LLMs

We thank the reviewer for their valuable feedback and comments.

Question: 1-The model is very similar to LLaVA: EMMA introduces CLIP-text encoder and an early fusion module.

Answer: Please refer to the first topic in the General Response, where we have thoroughly clarified the goals and design principles of EMMA.

Question: 2-The involvement of CLIP-text encoder is not very convincing: CLIP is trained for classification task only, but EMMA is a general-purposed instruction-aware MLLM. I am doubtful about the effectiveness when CLIP-text encoder deals with long instructions. You may make some visualization of the activation when CLIP encounter some long instructions / tasks apart from classification.

Answer: Thank you for discussing the CLIP's training objective: aligning corresponding image and text embeddings while maximizing the separation between unrelated pairs. It’s worth noting that the training dataset for CLIP comprises at least 400 million pairs, which extend far beyond simple classification descriptions (e.g., "A photo of a cat"). This extensive and diverse data enables the model to effectively learn embeddings that are semantically aligned across the two modalities. This cross-modality alignment is the reason that the CLIP visual encodings are preferred over other candidates to produce the visual encodings that are then passed through the LLM. In this work, we propose that leveraging the text encoder alongside the visual encoder to encode instructions allows for a better utilization of the two modules' alignment.

Due to the CLIP text encoder's input length limitation (a maximum of 77 tokens), we extract the most relevant portion of the instruction during both training and inference to feed into the text encoder. For example, given a full instruction such as: "A chat between a curious user and an artificial intelligence assistant. The assistant provides helpful, detailed, and polite answers to the user's questions. {question}" we isolate and input only the {question} into the text encoder.

Finally, to evaluate the performance of EMMA's modality adaptation module across various settings, it is helpful to refer to Tables 2, 3, and 4. These tables present results from over 10 benchmarks, containing diverse prompt structures.

Question: 3-The discussion in Sec. 3.2 should incorporate more datasets apart from MMVP. Because I am not sure of the effectiveness of the CLIP-text encoder as above.

Answer: Section 2 presents three key analyses: • General Analysis: This examines the weights of the modality adaptation module in a dataset-independent manner, providing insights into its overall behavior. • MMVP Dataset Analysis: This focuses on the MMVP dataset, which features similar images. It highlights the impact of instruction injection on the representations, enabling them to become more distinguishable. • Mutual Information Analysis: This explores the mutual information between visual and textual representations using another benchmark, LLaVA-in-the-Wild.

Question: 4-Unfair comparison: Table 1 without LLaVA, but Figure 3 only compare with LLaVA. Should compare LLaVA in Table1 and compare mPLUG-Owl2 and other methods in Figure 3. You may provide a table / figure to compare more methods.

Answer: Table 1 primarily focuses on comparing models with early modality adaptation modules; however, LLaVA has been included as suggested. Figure 3 (now it's Figure 5(a)) highlights a direct comparison between EMMA and LLaVA, as these two models share identical architecture and training setups, differing only in the modality adaptation module introduced by EMMA. Please refer to the first topic of General Response and Tables 2 and 4 for a comprehensive comparison between EMMA and many of the recent MLLMs.

Question: 5-The scale of Figure 3 is inappropriate, slight improves with significant distance, e.g. 64.3 -> 66.19 in MMB-en. I think a histogram bar will be better.

Answer: Thanks for your suggestion. We have adjusted the scales to accurately reflect the correct differences; please refer to Figure 5(a).

We thank the reviewer for their valuable feedback and comments.

Question: Inconsistent writing and presentation: The overall writing of the paper is somewhat disorganized, making it difficult to follow and reducing its readability. For example, Section 3.1 is intended to introduce the EMMA method, but the section ends with a discussion of experimental results (in the last paragraph on page 6). Similarly, in Section 3.2, which is supposed to focus on the Analysis on Modality Adaptation by EMMA, the section starts by explaining model details, which would be better suited earlier in the paper. Additionally, Figure 1 is referenced only on page 6, even though it appears on page 2. Many figures and tables also lack proper axis labels and detailed captions, which further hinders the reader's ability to understand the data. For instance, it would be helpful to clarify the contents of the axes in Figures 2, 5, and 6.

Answer: Thank you for your comments on the presentation. The revised version of the paper is now clearer and easier to read. The updates on the paper are listed in the third topic of the General Response.

Question: Unfair experimental comparisons: The paper does not provide a fair and direct comparison between different vision-language adapters, which is essential since the focus is on adapter efficiency. While achieving complete consistency between the vision and language models might be challenging, it would be beneficial to at least include an additional set of experiments that utilize other adapters, such as [1-2], alongside the same vision and language models. This would allow for a more fair comparison of their performance against the EMMA adapter. By keeping the vision and language models constant across different methods, the paper could offer a clearer evaluation of the specific contributions made by EMMA’s adapter design.

Answer: Thank you for your suggestion. MMA [1] is a concurrent work to EMMA and introduces a Multi-Modal Adapter that employs a cross-attention-based module in its architecture. However, the Multi-Modal Adapter proposed in MMA is specifically designed for classification tasks, and its capabilities as a VL (Vision-Language) adapter for MLLMs have not been explored. EMMA differentiates itself from MMA in two key ways:

• Innovative Alignment Approach: EMMA is the first to demonstrate that leveraging the inherent alignment between the visual and textual components of CLIP can enhance early modality adaptation in MLLMs, reducing the reliance on parameter-intensive modality adaptation modules.

• Architectural Differences: The architecture of MMA’s Multi-Modal Adapter is distinct from EMMA’s VL adapter, reflecting differing design philosophies and applications.

The following table presents a comparison of three MLLMs with identical setups, differing only in their modality adaptation modules: (1) LLaVA, the baseline model without a modality adaptation module, (2) EMMA, and (3) MMA. The results demonstrate that EMMA consistently achieves the highest performance across all benchmarks, except for MuirB, where MMA outperforms. MMA shows improvements in benchmarks such as SQA and MuirB compared to baseline, highlighting the strength of its modality adaptation capabilities. However, its complex architecture results in overfitting as reported for some benchmarks like GCA, MMB, MMVP, and A-Bench. We hypothesize that effective modality adaptation, refining visual representations through instruction encodings, requires a meticulously balanced design, particularly when working with an efficient training set of only 1.2 million samples (as the baseline). This ensures that the generalization of visual representations is maintained while seamlessly integrating instruction information.

The Tip-Adapter[2] method introduces a fine-tuning strategy for vision foundation models by incorporating an adapter module immediately after the vision model. This approach confines the fine-tuning process to the adapter module, keeping the vision foundation model frozen. Essentially, Tip-Adapter mirrors the approach used by LLaVA, where an adapter is added after the vision encoder and before the LLM component to preprocess visual encodings for the language model. However, Tip-Adapter does not propose a modality adaptation module or early fusion of text encodings with visual representations.

[1] MMA: Multi-Modal Adapter for Vision-Language Models
[2] Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

Question: Lack of comparison with visual-language early fusion works (e.g., QA-ViT[1]).

Answer: Thank you for bringing this paper to our attention. The QA-ViT [1] paper introduces a Question-Aware Vision Transformer approach for multimodal reasoning, embedding question awareness directly within the vision encoder. In contrast, our approach in EMMA does not modify the internal components of the vision encoder. Instead, we introduce a modality adaptation module that operates after the vision encoder. The key advantage of our approach over QA-ViT is that, by preserving the internal mechanics of the vision encoder, we maintain the generalization capabilities of the visual representations it produces. This ensures that the benefits of the pre-trained vision encoder's broad applicability are retained, providing a robust foundation for a wide range of tasks. Please check the following for a comparison between our method and QA-ViT, the results for QA-ViT are taken from their paper.

Traditional benchmarks:

As we can observe from the table above, on the traditional benchmarks, EMMA significantly outperforms QA-ViT by 8.92% and 19.98% on VQAv2 and VizWiz respectively while QA-ViT beats EMMA on TextVQA by a small margin. Also, the evaluations in the QA-ViT paper are limited to the traditional benchmarks and they haven't utilized the MLLM-specialized benchmarks.

[1] Question Aware Vision Transformer for Multimodal Reasoning. In CVPR, 2024

Question: In line 295, the claim that leveraging the inherent alignment properties of CLIP encoders obviates the need for complex cross-modal modules and extensive training to attain alignment is deemed unwarranted, given that the instruction embeddings fed into the LLM are not inherently aligned with CLIP features. Indeed, the proposed EMMA incorporates a projection layer akin to LLaVA after its Modality Adaptation module and also adopts the pre-alignment training from LLaVA. The reviewer considers this projection layer a pivotal component of EMMA's alignment achievement. Conducting additional ablation experiments wherein this projection layer is removed could potentially strengthen the authors' argument.

Answer: Please refer to the second topic (The direct effect of EMMA's modality adaptation) in the General response where we have addressed this concern.

Question: While Line 319 mentions the utilization of data from the same datasets, the 1.8M training samples employed by EMMA deviate in quantity from those used in any of the baselines, rendering it challenging to demonstrate the effectiveness of the Modality Adaptation module. Given that LLaVA 1.5 is fully open-source and shares a very similar structure with EMMA, adding the following experimental results would significantly strengthen this paper: (1) the outcomes of EMMA using the 1.2M training set from LLaVA 1.5, and (2) the results of LLaVA 1.5 utilizing the 1.8M training set from EMMA.

Please refer to the second topic (The direct effect of EMMA's modality adaptation) in the General response where there's a comparison between LLaVA-1.2M and EMMA-1.2M.

Following your suggestion we trained LLaVA-1.5 with 1.8M data and the results are provided in the following table, highlighting that EMMA significantly outperforms LLaVA on benchmarks that benefit from instruction-aware representations, such as VQAv2, VizWiz, SciQA, Muirbench, and hallucination-related tasks. While EMMA also performs better on other benchmarks with smaller differences, LLaVA slightly outperforms EMMA on the GQA benchmark. Overall, this demonstrates EMMA’s strength, particularly in tasks requiring effective alignment between visual and textual modalities.

We thank the reviewer for their valuable feedback and comments.

Question: In line 251, it is mentioned that current methodologies depend on complex cross-modality modules, particularly highlighting mPLUG-Owl2. However, the paper does not provide a theoretical comparison with methods like LLaVA and Qwen-VL, which utilize straightforward projection layers that add only a minimal number of parameters to achieve visual-language alignment.

Answer: In the paper, we compare our approach with both LLaVA and Qwen-VL, as detailed in Table 2 and Table 4, with a summary provided below for reference. Among the three methods, LLaVA[1] does not include a module for the early fusion of visual and textual tokens, deferring modality adaptation entirely to the language model (LLM) component. In contrast, EMMA incorporates a lightweight modality adaptation module with just 1.2M parameters, offering an efficient solution. On the other hand, Qwen-VL[2] utilizes a cross-attention-based adapter module with 80M parameters, making it significantly more complex and parameter-intensive compared to EMMA.

First set of benchmarks

Second set of benchmarks

As indicated in the tables, EMMA outperforms other models in 8 out of 10 benchmarks, while being equivalent as competing approaches for MMMU.

[1] Improved Baselines with Visual Instruction Tuning
[2] Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

Question: The proposed EMMA introduces a Modality Adaptation module based on the correlation between word tokens in instructions and visual tokens. However, this methodology lacks performance guarantees when instructions are verbose or correlate poorly with visual tokens. The benchmarks selected in the paper are skewed towards VQA tasks, which inherently possess a high correlation between instructions and visual tokens. It is imperative that benchmarks assessing fine-grained perception capabilities, such as document understanding and OCR tasks (e.g., Chartqa[1] and Docvqa[2]), should also be evaluated to ensure comprehensiveness.

Answer: Thank you for raising this question. In EMMA's design, we prioritized a lightweight architecture to preserve the generality of visual tokens and minimize overfitting. As a result, EMMA is expected to retain the performance of LLaVA, in scenarios where the instructions correlate poorly with the visual tokens and improve the performance when the visual tokens can be refined by leveraging the instruction tokens. The following table compares the performance of LLaVA and EMMA, demonstrating that EMMA achieves performance levels similar to LLaVA for these cases.

[1] Chartqa: A benchmark for question answering about charts with visual and logical reasoning.
[2] Docvqa: A dataset for vqa on document images