Streamline Without Sacrifice - Squeeze out Computation Redundancy in LMM

We appreciate your constructive and insightful feedback. We carefully considered your reviews and revised paper accordingly. Below, we directly address each point you raised:

• Incomplete evidence for the existence of computational redundancy
  Thanks for your suggestion to validate our observations on larger-scale models. Initially, we explored redundancy patterns using six widely used LLMs to broadly cover commonly studied architectures. Following your recommendation, we additionally conducted experiments on a larger-scale model, Qwen-2.5-32B (using LLaVA-1.5 style image encoding), and observed similar computational redundancy patterns (See updated Figure here). This confirms that our earlier findings generalize to larger models.
• Lack of in-depth analysis for the performance improvement
  Your suggestion to clarify the internal mechanisms underlying performance improvements is very valuable. To elucidate this, we measured the MIR (Modality Integration Rate) score [1], which quantifies alignment between visual and textual tokens (lower scores indicate better alignment). Evaluating 100 randomly sampled instances from the DetailCaps-4870 benchmark [2], we found that applying ProxyV-L12 to Vicuna-7B decreased the MIR score from 3.62 to 3.10, reflecting improved modality alignment due to decoupled vision modules. We will expand upon this insightful analysis in our revised manuscript.
• Internal relationship between task characteristics and algorithm performance
  Thanks for your constructive suggestion. Instead of defining tasks as fine-grained and coarse-grained based on human priors, we will provide quantitative metrics to represent task characteristics. Here we take ChartQA and GQA as two representative tasks, and we downscale the input image to 336*336 and measure the relative performance drop. ChartQA's performance drops by 28.7%, while GQA's performance drops by only 0.1%, indicating ChartQA requires much more fine-grained information than GQA. Then, we conduct experiments with the ATN+FFN (skipping both visual attention and FFN) variant to study the appearance of visual computation redundacy. We find that to retain the original performance, the skipping operation has to be applied after layer-17 for ChartQA and only layer-6 for GQA. This indicates that the visual computation redundancy appears from early layers on less fine-grained dataset GQA. We will thoroughly detail these results and analyses in the revised paper.
• Evaluation criteria may not be comprehensive enough
  Following your advice, we expanded our evaluations across additional MLLM benchmarks (see our response to Reviewer nPTg). We will provide results for all model variations in our revision.
• The results of applying ProxyV to different layers could be presented with more fine-grained details.
  We provide the full evaluation results on each benchmark in Section A of the Supplementary.
• Validate ProxyV with larger scale data & models
  Thanks for your valuable advice. We further increase the data scale to 3M and validate our method with Vicuna-7B (results shown in the table below) and observe a consistent conclusion. Due to computational constraints, we currently do not have results for even larger datasets or models such as 32B-scale models, but we plan to explore this in future work.

• Results of ATN+FFN
  In Table 2, the performance of ATN+FFN is much worse than ATN for the layer-0 case, achieves similar performance for the layer-12 case, and improves the performance for the layer-16 case. We have analyzed the reason in the last part of Section 2 and attribute the performance gain to the additional vision specific modules in ATN+FFN and the negative impact of additionally skipping FFNs becomes smaller in later layers. Our focus is to ensure no performance drop after the acceleration. Note that ATN+FFN still incurs performance degradion when applying from Layer 12 ($Score_{fine}$ = 54.28) and only achieves no performance loss applying from ($Score_{fine}$ = 54.85), while ProxyV-L12 achieves performance gain ($Score_{fine}$ = 55.16). In this case, ProxyV-L12 also has better efficiency than ATN-FFN-L16. All experiments are conducted in the same setting for fair comparison.

[1] Huang, Qidong, et al. "Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate." arXiv preprint arXiv:2410.07167 (2024).

[2] Dong, Hongyuan, et al. "Benchmarking and improving detail image caption." arXiv preprint arXiv:2405.19092 (2024).

Thank you very much for your thoughtful feedback. We have carefully considered your comments and made detailed revisions to address each point as follows:

• Generalizability of Claims
  The exploratory experiments presented in Section 2 are designed to identify computational redundancy patterns rather than propose universal claims or laws. Through our experiments on a wide range of LLMs, we can observe clear signals of computational redundancy on visual tokens, and we also explicitly state that "different models exhibit different patterns" in L130. Our intent was to highlight potential redundancy, motivating further focused studies. We also additionally add experiments on a deeper model, Qwen-2.5-32B (using LLaVA-1.5 style image encoding), and observed similar computational redundancy patterns (See the updated Figure here). We will revise our paper to make the claims more clear.
• Comprehensive Benchmarking and Evaluation
  We appreciate your suggestion regarding the comprehensiveness of evaluation benchmarks. Accordingly, we have expanded our evaluation to include multiple popular MLLM benchmarks. Our extended evaluation (detailed below) clearly shows that our method consistently achieves equal or superior performance compared to the baseline, while other token reduction methods show a notable gap in fine-grained visual grounding tasks like RefCOCO. We will provide additional comprehensive evaluations of all LLM variants in our next revision. Also, we clarify that we initially mainly focused on fine-grained benchmarks as they are good indicators for possible visual information loss, which is critical for MLLM acceleration methods.

• Unclear experimental details
  We have provided our experimental setting and details (data, training pipeline, model structure, image encoding scheme) for all experiments in Section 4 and provide more model hyperparameter details in Section B of the Supplementary. We will add more experimental and evaluation details in our revised version.
• Absolute values for FLOPs & Time, ProxyV’s additional costs
  Thank you for your valuable suggestion. We provide the absolute FLOPs & time values for Vicuna-7B/13B based models below, and we will add the results for all models in our revised version. All the FLOPs and times are directly measured in the same setting for all models, so the additional operations in ProxyV are already included. We will add clarification about this in the paper.

• Missing Essential References & Only compared with two non-SOTA competitors
  Thank you for recommending additional related papers. According to the ICML 2025 review instructions, "Authors cannot expect to discuss other papers that have only been made publicly available within four months of the submission deadline." PyramidDrop (CVPR 2025) is the most recent token reduction method that has been peer-reviewed to the best of our knowledge and can well represent the SOTA performance of token reduction methods. Besides, we emphasize that our proposed ProxyV approach is orthogonal and complementary to token reduction methods, with combined effectiveness demonstrated in Section 3.3.
• Figure font size and detailed results in tables
  We appreciate your suggestion on improving readability. We have modified our figures in this link. The detailed benchmark results for all experiments are provided in Section A of the Supplementary.

Thank you for your valuable feedback. We have reviewed your suggestions thoroughly and made corresponding revisions to address each of your concerns as outlined below.

• Paper Organization and Readability
  We appreciate your suggestion on improving readability. We have modified our figures, and the revised figures can be found in this link. We will reorganize the paper for clarity. We will add clearer captions, make expressions more consistent, and provide a more formal, detailed explanation of our approach to ensure better understanding.
• Experimental Details
  We have provided our experimental setting and details (data, training pipeline, model structure, image encoding scheme) for all experiments in Section 4 and provide more model hyperparameter details in Section B of the Supplementary. We will add more experimental and evaluation details in our revised version.
• Comparison on more MLLM benchmarks
  In response to your valuable comment on broader comparisons, we have expanded our evaluation to include additional standard benchmarks for MLLMs, as summarized in the table below. Our revised manuscript will include extensive evaluations across different LLM variants. From this table, we can see our method ensures no performance loss or achieves performance gain on most benchmarks, while token reduction methods have inferior performance on grounding benchmarks like RefCOCO, which requires finer visual information. Also, we clarify that we initially mainly focused on fine-grained benchmarks as they are good indicators for possible visual information loss, which is critical for MLLM acceleration methods.