Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models

1. The paper presents a method for enhancing visual understanding in MLLMs by focusing on visual details relevant to the textual prompt. While the concept of improving model performance through prompt-related visual cues is not entirely new, with prior works like API[1], AGLA[2], MAVEN[3], and ConrtolMLLM[4] exploring similar ideas. Despite some innovation in the design, the contributions do not seem to offer substantial new value or advancements in the field.
2. There is an inconsistency in the choice of baseline models. Line 353 mentions initializing parameters with MiniGPT-V2, yet Tables 2 and 3 use MiniGPT-4 as the baseline.
3. Ablation studies, particularly those involving parameter tuning (as shown in Table 7), should ideally be conducted on a validation set to ensure the generalizability of the findings. Although most researchers have currently overlooked this point.
4. Since Table 1 compares earlier models that utilize different visual encoders and language decoders, it would be beneficial to include comparisons with the latest methods that are not contemporaneous, such as LLaVA-Next. Additionally, it would be insightful to see if the authors' proposed method could be applied to other state-of-the-art models like LLaVA-Next or others.
5. It would be valuable to see comparisons with models that possess referring and grounding capabilities, such as Qwen-VL[5], Ferret[6], and ViP-LLaVA[7].
6. Figure 1 could be further refined to make it more concise and intuitive. Improving the figure to be more straightforward would enhance the reader's understanding of the concepts presented in the paper.

[1] Attention Prompting on Image for Large Vision-Language Models.

[2] AGLA: Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention.

[3] MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model.

[4] ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models.

[5] Qwen-vl: A frontier large vision-language model with versatile abilities.

[6] Ferret: Refer and Ground Anything Anywhere at Any Granularity.

[7] ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts.

The contributions of this work may not be as significant as the paper claimed.

First of all, given the recent efforts on the native multi-modal foundation model, such as EMU3, the adapter approach may be a temporary solution for MLLM.

The motivation to come up different ways to inject visual tokens to an LLM according to different prompts may not be that strong. The fundamental way to improve VQA shall be on how to generate detailed and accurate description and representation of image. Using the case in Fig.1 as an example, if MLLM can describe the image as “Along the pool in the backyard of a house, there is a table with 5 drinks on it and 4 chairs”, surely an LLM can conduct VQA and answer “Is there a pool in this image?” and “How may drinks are on the table?” without an adaptive adapter. Please send this photo to OpenAI o1 and check if it can generate a detailed description of the image. So it is arguable that do we really need to change the way to encode visual patches according to each textual prompt.

The technical contributions are a bit thin which are from line 253 to line 284 and Fig.3, while, most of the paper discusses the background and difference to previous works. The length of the paper could be trimmed, e.g., like Fig.1, the specific image is not informative, perhaps only keeping the same color tokens and different color tokens is enough.

The experiments verified the proposed schemes are useful to the VQA tasks on two datasets. What about its performance on more complicated tasks such as multi-round conversation w.r.t. images or video understanding.

Regarding to the writing, the paper tends to use some generic and vague expressions, like “understands information”, “to extract the most informative clues”. Btw: if your goal “aims to extract the most informative visual clues”, it does not mean you did “extract the most informative visual clues”, you still need to justify why you achieve this, in what sense it is “informative” (which is a very vague term), in what sense it is the “most” informative, etc.

1. The major weakness is the lack of novelty. Multimodal fusion for vision encoders has already been extensively explored and validated. Compared to VisionLLM and InstructBLIP, the proposed prompt-aware global and local attention is merely a combination of existing techniques, offering minimal contribution.
2. The experimental comparison in Table 1 is not fair. This paper uses LLaMA2-7B as the large language model, while its main counterparts, such as InstructBLIP, use the weaker Vicuna-7B model. Since the choice of vision encoder and LLM directly impacts multimodal performance, the performance improvements in the experiments can not reflect the advantages of the proposed method.
3. As shown in Eq. 5, compared to cross-attention, the proposed prompt-aware local attention uses global softmax to create a single association between text tokens and vision tokens. This implicitly assumes that each text token will have a high correlation with only one visual token. However, this assumption does not hold in many cases, especially when visual patches are densely divided.

• The motivation of dual attention is not clear. The visual clues can be captured by cross-attention at token level from prompt. Why do we need to split it into global attention and local attention? And the visual clues are typically given at a word/token level. Why do we need a global attention?
• The differences in the metrics in the "Calculate" column of Table 3 are quite substantial and require further explanation. The Q-Former∗ + Linear model achieves a score of 78.07, while the proposed method only scores 50.
• There are two bolded numbers in the "Position" column of Table 3. Please correct this issue.
• The results in Table 1 indicate that the proposed method performs poorly on the color recognition task (99.99 vs. 165.00). What factors might contribute to this discrepancy? Could you provide an appropriate analysis or evidence to support this observation?