Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers

1. This paper repeatedly emphasizes that the main limitation of FastV lies in its reliance on attention maps, which restricts it to using only the simplest attention implementation. However, this is fundamentally an engineering issue related to code implementation and does not merit extensive discussion in the paper. There are two straightforward solutions to address this: (1) set only the second layer’s attention to “eager” mode while employing flash_attention_2 for the remaining layers, or (2) store the key and value states of the second layer and recompute the attention map with (Lt, Lv) shape. Both methods can significantly reduce latency and flops. Therefore, all data related to FastV in the experiments should be reconsidered, particularly the comparison of training times in Section 5.3, where setting all FastV layers to eager mode is highly unreasonable, making the training time reported for FastV invalid.
2. When conducting research on efficiency, the primary focus should be on performance. Ideally, the efficiency strategies we aim to achieve should provide the greatest possible acceleration while maintaining comparable performance levels. However, this paper lacks any tables presenting the specific benchmark scores, relying solely on line graphs, which makes it difficult to assess the exact performance degradation. Moreover, the latter half of the line graphs has limited practical significance; for instance, when the benchmark performance drops by around 4%, the actual performance decreases far more, rendering such acceleration strategies unusable. It would be beneficial to select a few settings and display the key benchmark scores, such as for TextVQA, GQA, and SEED-Bench.
3. When comparing inference acceleration, this paper uses FastV as a major baseline. However, FastV is a training-free method, whereas the approach proposed by this paper requires complete training process before inference acceleration is possible. Therefore, this comparison is unreasonable from a cost perspective.
4. The reported benchmark performance of fastv after training was even lower than that of inference-only. Based on experimental experience, this result appears highly unreasonable, suggesting a review of the code for potential issues, and there is reason to suspect the baseline may have been artificially lowered.
5. Please provide a simple baseline, just pooling after the first layers to retain visual tokens. According to the paper’s methodology, retained visual tokens are intended to aggregate information from other image tokens. If using learnable tokens is feasible, a simple and straight approach would be to directly leverage part of the original image tokens to gather information.
6. In Section 4.1, it states, "We use the AdamW optimizer (Loshchilov & Hutter, 2017) with a learning rate of 0.0001 and no weight decay for one epoch." I noticed that this learning rate differs from the original LLaVA's rate of 0.001, indicating that you have adjusted the training hyperparameters. If this method were to be applied to another model, it would also require extensive hyperparameter tuning, which could incur significant computational costs.

1. The paper aims at improving the efficiency of the VLMs. I think more baselines should be incorporated to demonstrate the effectiveness of the method. Some examples are: [1] PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling [2] LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference [3] Efficient Inference of Vision Instruction-Following Models with Elastic Cache

2. Also, I think more methods should be discussed in related work section, about the development of efficient VLMs, currently still only fastV is mentioned.

3. Some questions in the experiments. [1] why using 'zero' or 'image' can still maintain the performance to some extend as shown in Figure 16. I do not really get why zero can do something. [2] I notice that some benchmarks, such as scienceqa, even perform much better with only 8 visual tokens. Can you provide some explanations?

Due to the questions above, I think some revisions are needed, including more baseline comparisons, revision of the related work section and more explanations on experiment results. Still, I will consider raising my score if my questions are well-solved.

This paper’s weaknesses are mainly discussed in Sec. 6

1. Dependence on Training Integration: The method is not training-free, which may limit its use for pre-trained models that cannot be modified or retrained.
2. Limited Adaptability of Visual Registers: The performance drops when the number of visual registers is adjusted post-training. The paper acknowledges this and suggests potential auxiliary loss functions as future work but does not address it in the current method.

Despite these weaknesses, the paper's contributions remain impactful, offering a compelling method that balances efficiency and performance effectively.

1. Since the advantages of additional register are not adequately presented. It's hard to attribute the superiority of the experimental results to the proposed method.
2. It is advisable to add a baseline experiment that directly uses fewer visual tokens. For instance, a pooling layer could be applied directly after the image tokenizer to achieve a similar computation FLOPs.

1. The approach to visual token summarization feels overly simplistic. Prior studies suggest that fewer visual tokens are often sufficient, and Victor’s method lacks a more sophisticated or in-depth design for how register tokens are used to retain critical information. The current version seems to only provide a simple and straightforward baseline, while an exploration of more refined register token strategies could potentially minimize performance drops further or increase acceleration gains, maybe reducing >43% training time. For example, an advanced token-drop strategies, the relationship of register token drop ratios with increasing layer depth should be explored further.
2. Some comparisons with visual token compression models are missed. Victor’s efficiency claims would be more compelling if compared against models that perform early token compression before inputting to the LLM, such as LLaMA-VID. LLaMA-VID uses only two visual tokens to achieve core functionality, this paradigm seems to be also or even more efficient in theory.
3. While Victor is evaluated using different LLM architectures, it relies on only the LLaVA architecture for vision-language fusion. Comparisons with alternative VLM architectures, such as Mini-Gemini, Cambrian-1, or InternVL, would have offered a more robust evaluation.