VCM: Vision Concept Modeling with Adaptive Vision Token Compression via Instruction Fine-Tuning

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1: Heuristic-Based Concept Length Estimation.

Response:

To investigate the impact of estimated length versus precise length on performance, we conducted the following comparative ablation experiments. Specifically, we trained the model for 500 steps using 5K samples (as mentioned in line 85 of the paper), which were labeled by GPT-4o.

---

As shown in the table, the experimental results under precise length and estimated length settings are quite similar. This is attributed to VCM's inherent error-tolerance mechanism:

1. Handling underestimated lengths: When the estimated length is shorter than ideal, VCM compensates by merging more vision tokens into a single vision concept token, thereby increasing the average information capacity of each vision concept token.

2. Handling overestimated lengths: When the estimated length is longer than necessary, VCM generates longer vision concept tokens, ensuring that the subsequent LLM can still extract sufficient information for downstream tasks.

W2: High Dependency on Keyword Selection.

Response: The keyword selection module in VCM demonstrates reliable performance in selecting relevant keywords. Additionally, VCM's random masking strategy introduces error tolerance and robustness against incorrect keyword predictions. Together, these two components ensure efficient and accurate learning, thereby reducing the dependency on keyword selection. To separately validate these two aspects, we conducted the following experiments:

To better illustrate the reliability of the keyword selection module, we randomly selected 1K samples from the 5K samples (as mentioned in line 85 of the paper) labeled by GPT-4o and computed the success rate of keyword selection by the module during training. The experimental results are reported in the table below.

As shown in the table, during pretraining, the keyword selection module progressively learns the relationships between images and text, ultimately achieving a 94.3% success rate in selecting the correct keywords.

To further validate the robustness of VCM, we simulated unreliable predictions by the keyword selection module during inference. Specifically, we randomly replaced selected keywords with non-keywords at different replacement probabilities $r$, and then tested the model while recording the average visual concept token length.

As shown in the table, VCM exhibits strong error tolerance. When the keyword selection module makes incorrect predictions, VCM tends to generate longer visual concepts to ensure that the subsequent LLM can access sufficient prior information for downstream tasks.

W3: Lack of Analysis on VCM's Own Overhead.

Response:

We sincerely thank the reviewers for their valuable suggestions, which have significantly improved the quality of our work. In response, we have added the following experiments to provide more detailed insights into the latency overhead of VCM components and compared them with other methods. Specifically, we separated the latency contributions of the original LLM and the additional modules introduced by each method for more precise comparisons.

As shown in the table, two key observations can be made. First, the non-LLM overhead introduced by VCM is only slightly higher than FastV. This is because our method utilizes a lightweight design consisting of only a two-layer attention module, resulting in no significant increase in parameter count compared to previous methods. Second, our approach incorporates weighted averaging and parallel optimization techniques, as detailed in Appendix Algorithm 2, which further reduces the latency caused by the additional components in our method.

W4: High Architectural Complexity.

Response:

We sincerely thank the reviewer for their careful review. In fact, while VCM may appear theoretically complex, in practice it consists of only two attention layers, resulting in no significant increase in parameter count compared to previous methods. Additionally, its computational complexity remains the same as earlier approaches, at $O(M^2)$. Moreover, we have implemented parallel optimization techniques, enabling VCM to operate efficiently.

As a plug-and-play method, VCM has consistently demonstrated its simplicity, efficiency, and versatility across diverse model architectures (e.g., LLaVA [1], QwenVL [2]) and various application scenarios (e.g., standard images, high-resolution images, and video). This further highlights its generalizable nature and adaptability for different use cases.

W5: Computational Analysis Deferred to Appendix.

Response:

We sincerely thank the reviewer for their thoughtful suggestions, which have helped improve the readability of our work. Due to page limitations, we chose to place the performance comparisons in the appendix. In the revised version, we will include a dedicated discussion and highlight the location of the relevant tables to ensure readers can easily find and focus on them.

Q1: Could the authors provide a more detailed explanation for this counter-intuitive result?

Response:

Using fewer tokens yielding better performance on certain benchmarks is a normal phenomenon and has been observed multiple times in previous works, such as MQT-LLaVA [1] (Table 1, MMB), FastV [2] (Table 1, MMMU), PDrop [3] (Table 1, POPE), and VisionZip [4] (Table 11, VizWiz). This arises because different tasks demand varying levels of perceptual granularity from the model.

For instance, tasks like VizWiz emphasize local perceptual abilities, where excessive tokens may introduce noise and redundant information, leading to reduced performance compared to fewer tokens. Conversely, tasks like MME rely heavily on global details, where more tokens provide greater benefits, making such phenomena less likely to occur. This difference in task requirements explains the seemingly counterintuitive results observed for certain benchmarks.

Q2: Fine-tuning the vision encoder led to slightly lower average performance than keeping it frozen, which is unusual.

Response:

For LLaVA and subsequent works [3,4,5,6] based on LLaVA, freezing the vision encoder has become the default configuration. This is not due to increased computational overhead—since the vision encoder's computation cost is negligible compared to that of large language models—but rather because experiments have shown that jointly training the vision encoder with low-resolution images and limited data often yields suboptimal results.

A commonly accepted explanation within the community is that freezing the vision encoder helps preserve the pre-trained knowledge in the encoder, preventing its disruption when trained on small datasets. To ensure fair comparisons, we also report results for models with the vision encoder unfrozen in Table 1. As can be observed, unfreezing the vision encoder parameters indeed leads to suboptimal performance, particularly for the baseline LLaVA-v1.5 model.

[1] LLaVA: Large Language and Vision Assistant

[2] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

[3] MQT-LLaVA: Matryoshka Query Transformer for Large Vision-Language Models

[4] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

[5] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction

[6] VisionZip: Longer is Better but Not Necessary in Vision Language Models

Thank you again for your insightful comments. If you have other comments, we are happy to address them to polish this work. We look forward to contributing to the development of both the multi-modal research and the open-source community.

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1.1: The length estimation employs min-max normalization with a fixed domain-specific threshold τ, lacking sensitivity analysis; whether task-specific τ values are required for different tasks remains unaddressed.

Response:

We hypothesize that the fixed domain-specific threshold $\tau$ refers to the information domain scalar $S$. To investigate the impact of the dynamic information domain $S$ on VCM, we categorized the training data. From the results in the ablation study shown in Table 4, we observed that for localization and fine-grained perception tasks, a smaller information domain yields better results. Conversely, for tasks requiring global information, a larger information domain is more effective.

Based on this observation, we categorized the instruction-tuning data by task type. For open-ended tasks, we set the information domain to $1/2$, while for fine-grained perception tasks like the RefCOCO localization task, we set the information domain to $1/4$. We then conducted mixed training for 500 steps, and the experimental results are presented below.

As shown in the table, using mixed training with different information domains for different tasks did not lead to further performance improvements. We believe this is because dynamic information domains introduce challenges for the forward-backward optimization algorithm, making it difficult for VCM to discriminate whether the varying learning lengths are caused by differences in information domains or differences in keyword count. This conflict ultimately led to degraded performance.

Compared to dynamically altering the information domain during the training phase, we believe that masking keywords to adaptively select task-specific lengths during inference could be a more flexible and efficient approach.

W2: The figure of the overall framework is not clear enough. I suggest you can bind G^T,G^V..., with the corresponding tokens in the figure.

Response:

We sincerely thank the reviewers for their detailed and thoughtful feedback, which has greatly improved the readability and completeness of our work. In the revised version, we will incorporate binding to make the definitions more precise and clearer.

Q1&W1.2: Lack the effectiveness of direct token selection learning without a keyword module (KM).

Response:

We conducted a complete retraining of VCM using only vision token selection configurations and then performed comparative experiments under the condition of 144 tokens. The experimental results are presented below.

As shown in the table, relying solely on vision tokens for keyword selection prevents the model from leveraging the latent space of large language models, thereby limiting its ability to search for beneficial prior information. Furthermore, this approach fails to establish a robust visual concept model, making it unsuitable for dense perception tasks beyond VQA, such as RefCOCO. Consequently, its applicability is significantly restricted.

Q2&W1.3: What is the average VQA score of MQT-LLaVA under the 144-token condition? Does VCM's advantage stem from differences in search space or from textual priors?

Response:

We sincerely thank the reviewers for their thorough review. As presented in Table 1 of the manuscript, we have already provided a performance comparison between MQT-LLaVA and VCM under the 144-token condition (58.6 vs. 60.8). It can be observed that VCM demonstrates a clear advantage. We believe this stems primarily from two key factors:

1. Enhanced utilization of textual priors: VCM benefits more effectively from the latent space of textual priors, enabling better alignment between visual features and textual features.

2. Higher information density: At equivalent token lengths, visual concepts carry richer information content compared to visual tokens, thereby representing more information and further boosting performance.

Q3: Under 1024×1024 high-resolution images and 30 fps video streaming conditions, what are the comparative metrics of VCM versus VisionZip and SparseVLM in terms of GPU latency and memory consumption?

Response:

We sincerely thank the reviewers for their suggestions. In response, we provide a comparison of VisionZip and SparseVLM with VCM regarding FLOPs and GPU memory across high-resolution scenarios (LLaVA-Next) and video streaming (Video-LLaVA), as shown in Appendix Tables 7 and 8. From the tables, it is evident that VCM achieves the best overall results while maintaining competitive performance in both speed and memory consumption.

This advantage can be attributed to the following three reasons:

1. Model Parameter Efficiency: VCM employs a simple two-layer attention module, which introduces minimal additional parameters compared to previous methods such as SparseVLM and VisionZip.

2. Computational Complexity: VCM utilizes a forward-backward optimization algorithm, ensuring its computational complexity remains equivalent to methods[1,2,3,4] based on attention mechanisms at $O(M^2)$.

3. Parallel Optimization: VCM incorporates parallel optimization techniques, as detailed in Algorithm 2 of the appendix, significantly enhancing its runtime efficiency.

We believe these attributes make VCM highly effective and efficient for both high-resolution and video-streaming scenarios. Thank you again for your constructive feedback, which has helped us emphasize these strengths more clearly in the revised version.

[1] MQT-LLaVA: Matryoshka Query Transformer for Large Vision-Language Models

[2] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

[3] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction

[4] VisionZip: Longer is Better but Not Necessary in Vision Language Models

Thank you again for your insightful comments. If you have other comments, we are happy to address them to polish this work. We look forward to contributing to the development of both the multi-modal research and the open-source community.

Dear Reviewer u1dV:

We greatly appreciate the time and effort you dedicated to reviewing our paper. We have carefully addressed all your insightful suggestions and clarified any ambiguous points to improve our work. As the deadline for the discussion is nearing, could you kindly reconsider your evaluation based on the revised version? We are open to any further queries you might have and are eager to provide any additional information needed.

Thank you for your understanding and support.

Best regards,

Authors

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1: Strong dependence on the text

Response: The keyword selection module in VCM provides reliable keyword selection results, while the random masking strategy integrated into the VCM algorithm introduces error tolerance and robustness toward incorrect keyword predictions. Together, these two features ensure efficient and accurate learning, reducing the dependency on keyword selection. To verify these two aspects, we conducted the following experiments:

To better illustrate the reliability of the keyword selection module, we randomly selected 1K samples from the 5K samples mentioned in line 85 of the paper, labeled by GPT-4o, and calculated the success rate of the module in selecting the correct keywords during training. The results are summarized below:

As shown in the table, as training progresses, the keyword selection module effectively learns the relationships between text and images, ultimately achieving a 94.3% accuracy in selecting correct keywords.

To further validate VCM's robustness, during inference, we randomly replaced selected keywords with non-keywords at varying replacement probabilities r to simulate unreliable predictions by the keyword selection module. We then tested the model and calculated the average visual concept token length across SciQA, POPE, and MME datasets. The results are summarized below:

From the table, it can be observed that VCM’s masking strategy ensures strong error tolerance. When the keyword selection module produces incorrect predictions, VCM tends to generate longer visual concepts to ensure the downstream LLM receives sufficient prior information for task reasoning.

W2&Q1: Dynamic programming introduces O(M2) complexity for sequence alignment, which potentially limits scalability to long videos. This paper focuses on static images/videos, it is unclear how VCM would handle temporal dynamics in complex videos?

Response:

Compared to previous methods based on attention mechanisms[1,2,3,4], VCM has the same theoretical computational complexity of O(M^2), with no significant difference in this regard. However, VCM can be effectively combined with downsampling techniques, where long videos are first downsampled and then further compressed using VCM. This combination allows VCM to be efficiently applied to long video understanding tasks. To further validate the scalability of our method in complex long video scenarios, we selected the challenging long-video benchmark dataset, Video-MME, for evaluation and comparison. Specifically, we processed 64 frames per input, and the experimental results are illustrated in the figure below.

Compared to the baseline method Video-LLaVA, VCM reduces the input sequence length for a single frame from 256 to 17, while maintaining strong performance on the Video-MME benchmark. This demonstrates that our method, like previous approaches, is plug-and-play and exhibits excellent scalability.

Q2: Have you tested VCM on non-English datasets? How does reliance on English keyword priors affect performance in other languages?

Response:

To investigate the impact of non-English data on VCM's generalization capability, we collected the LLaVA-ZH Chinese dataset. We mixed it with the original LLaVA dataset and trained the model for 500 steps on the combined dataset. The model's performance was then tested on MMB-ZH and MMB-EN, and the experimental results are presented below.

As shown in the table, VCM can successfully train and test on Chinese data, achieving performance comparable to that on English data. This demonstrates VCM's strong cross-linguistic generalization capability.

Q3:Does VCM provide mechanisms to detect hallucinations or uncertain predictions when concepts are compressed (e.g., via confidence scores)?

Response:

We conducted an ablation study to evaluate the impact of different confidence score thresholds on model performance during conditional reasoning after training for 500 steps, where the thresholds were also used to detect and filter uncertain information, influencing the model's performance.

As shown in the table, increasing the confidence score threshold during inference helps filter out noisy and uncertain information, thereby improving the model's performance on downstream tasks. This indicates that the confidence scores in our end-to-end optimization design carry practical physical significance, aligning with the design principles stated in line 176 of the paper. Specifically, confidence scores represent the uncertainty in concept compression and can be effectively utilized to design mechanisms for detecting hallucinations or uncertain predictions.

[1] MQT-LLaVA: Matryoshka Query Transformer for Large Vision-Language Models

[2] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

[3] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction

[4] VisionZip: Longer is Better but Not Necessary in Vision Language Models

Thank you again for your insightful comments. If you have other comments, we are happy to address them to polish this work. We look forward to contributing to the development of both the multi-modal research and the open-source community.

Dear Reviewer UGcq:

Thank you again for your time and valuable feedback on our paper. We noticed that your official comment on OpenReview is still pending. Could you please let us know if there’s any additional information needed from our side? We’d greatly appreciate it if you could submit your comment when you have a moment.

Sincerely,

Authors

Dear Reviewer UGcq,

I hope this finds you well. As we near the end of the discussion period, we wanted to thank you once again for your thoughtful review. Please let us know if there are additional questions or aspects of the work you’d like us to address.

We highly value your expertise and feedback, and we’re eager to continue discussing any unresolved points that could help refine and improve our research.

Thank you for your time and for contributing to the review process.

Sincerely,

Author

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1: The notations are complicated.

Response:

Your careful attention to the consistency and clarity of the notation has been invaluable in helping us improve the quality and readability of our work. We fully agree with all your suggestions and are grateful that you took the time to identify these important improvements.

In the revised version, we will: (1) unify $f_{KS}$ and 'MHSA' in line 102 and Equation (2) to use $f_{KS}$ consistently; (2) simplify $f_{V2T}$ to $f_M$ in Section 3; (3) add a clear definition for the text prior $T$ in line 121 (confirming it refers to keywords); (4) fix the typo in line 121; and (5) use $f_T$ instead of $X^{\top}$ to represent the LLM's word embeddings for better notational consistency.

We are committed to implementing all these changes carefully. Thank you for your constructive feedback.

Q1: What is the objective of the binary classification in equation (3)? How to train the linear $f_{CLS}$?

Response:

Thank you for your comments. In Section 3.3, we have provided a detailed explanation of the training process. Here, we would like to further elaborate on the specific steps involved.

The binary classification in Equation (3) generates binary logits $Y^V$, which indicate whether each visual token should be retained. These logits are combined with the target sequence $Z^V$—obtained by expanding the token length using the prior distribution patterns illustrated in Figure 2—and subjected to optimization through the forward-backward algorithm. During this end-to-end training process, the linear classifier $f_{CLS}$ is trained to effectively learn the mapping between visual tokens and their likelihood of being selected.

Q2: How to obtain each probability $p(z_l|y_t)$ when conducting forward and backward probability transition in section 3.3?

Response:

Thank you for the feedback. Here, we provide further clarification.

Firstly, the logits $Y^V =$ y_t^V $^M_{t=1}$ are produced as outputs from the linear classifier $f_{CLS}$. Then, the probabilities $p(z_l^V | y_t^V)$ are computed using $p(z_l^V | y_t^V) = \text{Softmax}(Y^V)$. Essentially, this represents a binary classification probability since $z_l^V \in$ \star, \emptyset $$.

We will incorporate this explanation into the paper to make the presentation clearer and more accessible. Thank you once again for your insightful comments, which have been instrumental in enhancing the clarity of our work.

Q3: Please explain the implicit contrastive sampling module and the corresponding training process. These content seem missing in the main text and only appear in the caption for Figure 3.

Response:

As discussed in lines 125–130 of the main text, this module fundamentally serves as a data augmentation mechanism and does not contain trainable parameters. Specifically, we randomly apply masking to the extracted keywords, and VCM leverages the masked data to generate extended visual concepts. This ensures that the subsequent LLM processing does not encounter hallucinations or conflicts due to missing information.

In Table 4, we conducted ablation experiments to validate this masking strategy. The results demonstrate that this approach introduces greater data diversity without adding additional risks, significantly enhancing the effectiveness of VCM’s training and improving its final performance.

Additionally, we will include references to Figure 3 in the revised version to ensure smoother readability and make the explanation more intuitive for readers. Thanks for your valuable insights, which have helped us refine and improve the clarity of our work.

Q4:How does the proposed method address the first key challenge mentioned in line 73?

Response:

We sincerely appreciate the reviewers' valuable feedback. Here, we provide further clarification on this point.

By utilizing VCM's forward-backward optimization algorithm, we can efficiently perform self-supervised learning on large-scale VQA datasets that lack fine-grained annotations. This process allows us to construct a robust visual concept model. Furthermore, we employ the implicit contrastive sampling module as a data augmentation technique, which further increases the diversity of the training data and enhances the effectiveness of visual concept modeling.

With these two contributions, we address the first key challenge mentioned in line 73. Unlike previous methods, which require fine-grained annotations—not only identifying the keywords in the text but also aligning each keyword with specific regions in the image-our approach eliminates the dependency on such detailed labels, making it both more scalable and efficient.

Thank you again for your insightful comments. If you have other comments, we are happy to address them to polish this work. We look forward to contributing to the development of both the multi-modal research and the open-source community.