SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs

We sincerely thank the reviewer for their positive feedback on our paper, particularly for recognizing its clarity, the simplicity and intuitiveness of our proposed method. We greatly appreciate these encouraging comments. In the following, we will address each of the reviewer’s concerns and suggestions point-by-point to further clarify and strengthen our work.

Q1: The novelty of the proposed method is limited. The "saliency" is just the attention score which has been widely adopted in previous methods. Besides, I think the proposed "coverage" measure is quite similar to the diversity, which is also calculated with the cosine similarity, such as Divprune [1].

A1: Thank you for your comment. We respectfully clarify that our method is distinctive from existing works, with the following contributions:

1. Coverage is different from diversity: While both coverage and diversity involve cosine similarity, they serve fundamentally different purposes. For example, DivPrune [1] focuses on selecting tokens that are mutually dissimilar, aiming to maximize pairwise diversity among selected tokens. In contrast, our coverage-based selection encourages selecting tokens that best represent the entire token set by maximizing set coverage gain. As illustrated in Figure D1, DivPrune would select token C to maximize diversity from already selected tokens, while our coverage strategy selects token B because it better covers the unselected tokens A and C (i.e., provides broader representativeness). This distinction is also supported by empirical results, for example, our coverage-only baseline achieves 56.3% on TextVQA and 60.8% on MMB, outperforming DivPrune’s 54.9 and 59.3%, respectively.

Figure D1: Illustration of token selection. ● and ○ selected and unselected tokens.

  ●   ● 
●       ●    ○ A    ○ B    ○ C
    ●               ↑      ↑
                Coverage Diversity  

2. We are the first to jointly consider saliency and coverage: While saliency (e.g., attention scores) has been used in previous token pruning methods, these approaches often lead to semantic incompleteness by focusing only on highly attended regions. In contrast, our method, SCOPE, formulates token selection as a joint optimization over both saliency and coverage. This ensures that selected tokens are not only important (salient) but also representative of the full image (coverage), leading to improved semantic completeness and task performance.

We hope this clarifies the novelty of our approach. We will further improve the explanation and emphasize this distinction in the final version.

Q2: The proposed method should be evaluated on SOTA models to demonstrate the effectiveness, such as qwen2-vl.

A2: Thank you for the suggestion. Following your advice, we have evaluated our method on the Qwen2-VL model. The results are summarized in Table D1. As shown, our method achieves 94.6% of the full-model performance when retaining only 25% of the tokens. Furthermore, our method significantly outperforms prior approaches such as DivPrune, with a 3.7% improvement in average score under the 10% token ratio.

We will include these results in the final version to demonstrate the robustness and generality of our approach on strong vision-language models.

Table D1: Results on Qwen2-VL. The token ratio means the ratio of retained tokens.

Q3: Besides, could the proposed method be combined with LLM-based pruning methods, such as FastV?

A3: Thank you for the valuable suggestion. We have explored the possibility of combining our SCOPE method with LLM-based pruning approaches such as FastV. In particular, we first apply SCOPE to prune visual tokens before feeding them into the LLM, and then apply FastV for further pruning within the language model. The results are provided in Table D2. These results indicate that SCOPE can be effectively combined with FastV to further reduce the number of visual tokens while maintaining reasonable performance. However, there remains significant room for improvement, and we consider this a promising direction for future exploration.

Table D2: The results of SCOPE combined with FastV. We first retain 192 tokens and then prune R ratio tokens at K-th layers of LMM using FastV.

Q4: The proposed method should also be tested on video benchmarks to prove the validity.

A4: Thank you for the suggestion. Following VisionZip, we conducted experiments on the Video-LLaVA. The results are reported in Table D3. As shown, our method achieves the best performance among all compared methods. Surprisingly, even with aggressive pruning, our method almost fully preserves the original performance. This demonstrates the strong effectiveness of our method on video-language tasks. These findings also suggest that video benchmarks contain substantial redundancy, and token pruning has great potential for accelerating video LLMs without sacrificing performance. We will include these results in the final version.

Table D3: Results on video benchmarks.

Q5: I am interested in some OCR benchmarks, which demand more advanced token pruning methods for maintaining the performance.

A5: Thank you for your suggestion. In the main paper, we have already evaluated our method on several OCR-related benchmarks, such as MME and MMVet. To further demonstrate the effectiveness of SCOPE, we conducted additional experiments on more OCR-specific benchmarks. The results are shown in Table D4. As illustrated, our method consistently preserves performance and outperforms VisionZip across different token counts. This further supports the robustness of our approach for OCR tasks. We will include these results in the final version.

Table D4: More results on OCR benchmarks.

Q6: Why compute the SCOPE score with the product of visual attention score and coverage gain? Have you ever tested adding these two values? Is there an ablation study?

A6: Thank you for the insightful question. Here we compare these two designs:

• product promotes the selection of tokens that are both informative (salient) and non-redundant (bringing new coverage). This multiplicative formulation naturally down-weights tokens that are high in only one of the two dimensions.
• additive could overemphasize either highly salient but redundant tokens or low-saliency tokens with marginal coverage contributions.

We conducted an ablation study to compare the product and addition strategies. As shown in Table D5, the product-based formulation consistently outperforms the additive one across all benchmarks, demonstrating its effectiveness.

Table D5: Comparison of different combinations of saliency and coverage.

Second Invitation for Further Discussion

Dear Reviewer 6ZPM:

Given the review timeline and policy, we would be grateful for any indication of whether our responses have satisfactorily addressed your concerns, or if further clarification is needed.

We remain committed to improving our manuscript based on your expertise.

Thank you for your time and consideration.

Best regards,

The authors

Gentle Reminder: Reviewer Discussion Participation

Dear Reviewer 6ZPM,

Thank you again for your time and effort in reviewing our submission.

Just a gentle reminder that the discussion phase will end soon. We truly appreciate your earlier review, and would be grateful for any additional comments or clarifications you might share during this phase.

As noted in the NeurIPS policy, submitting the “Mandatory Acknowledgement” does not replace participation in the discussion. Your engagement would help ensure a more thorough and balanced evaluation.

Please let us know if we can clarify any points from the rebuttal or provide further information.

Best regards,

The Authors of Submission 4718

The author provided the rebuttal. Could you please check if the concerns are fully addressed?

Thanks, AC

Clarification Request Regarding Remaining Concerns

Dear Reviewer 6ZPM,

Thank you again for your thoughtful review and for the time you've dedicated to evaluating our submission.

As per the NeurIPS 2025 discussion guidelines, reviewers are encouraged to communicate clearly whether their concerns have been addressed:

“If authors have resolved your questions, do tell them so.”

“If authors have not resolved your questions, do tell them so too.”

We have provided detailed responses to the concerns you previously raised. We kindly ask whether there are any remaining issues you feel have not yet been addressed.

Just a gentle reminder that the discussion phase will end soon. We would truly appreciate any additional comments you might have.

Best regards,

The Authors

We sincerely appreciate the reviewer’s recognition of the core contribution of our work, i.e., the joint modeling of saliency and coverage to preserve semantic completeness. We are encouraged that both our visualizations and experimental results were found to effectively support this goal. We address your questions as follows:

Q1: Due to the introduction of an additional module, how much computational and time cost does SCOPE introduce compared to previous methods?

A1: Thank you for your question. As shown in Table 4 of the main paper and detailed again in Table C1 below, we analyze the efficiency of our method on LLaVA-NeXT 7B. Although SCOPE introduces slightly higher latency than PDrop, it still achieves a 3.0$\times$ speedup compared to the full-token baseline. Importantly, SCOPE maintains strong performance on the POPE metric (83.0% vs. 86.4%), showing that our token selection strategy preserves semantic completeness. In contrast, PDrop exhibits a significant performance drop (53.2% POPE), likely due to its reliance on text-visual saliency-only pruning, which may overlook tokens that are not highly attended but are semantically crucial.

Table C1: Efficiency analysis of our method on LLaVA-NeXT 7B. The experiments are conducted on a system equipped with $4\times$A100. $\Delta$ denotes the reduction ratio.

Q2: Table 3 shows that the increase in the ablation experiment is limited. Could author analyze the reason for this?

A2: Thank you for the question. We analyse the ablation experiments as follows:

1. SCOPE achieves significant gains compared to the random and saliency-only baselines, especially on challenging benchmarks such as POPE and TextVQA. This confirms the importance of informed token selection.
2. The coverage-only variant achieves very strong performance, validating our key design motivation: ensuring diverse and complete visual semantics is crucial for multimodal reasoning.
3. Saliency provides complementary benefits. Although its standalone performance is weaker than coverage-only, incorporating it into our full method (SCOPE) consistently leads to further improvements. This suggests that saliency helps refine the focus toward more informative regions that coverage alone might overlook.

In summary, SCOPE effectively integrates both saliency and coverage. The relatively small gain over coverage-only reflects the strength of the coverage, rather than a limitation of our approach. The consistent improvements across all benchmarks highlight the robustness and synergy of the combined design.

Thank you for your recognition of our work and the constructive feedback. We would like to address your concerns individually.

Q1: The idea of using set coverage to guide pruning is novel to me, but the idea of token similarity is not novel and can be found in some token merging work, such as [1].

A1: Thank you for recognizing the novelty of using set coverage in our pruning framework. We acknowledge that token similarity has been widely adopted in previous token compression approaches, including token merging methods such as [1]. However, our work differs fundamentally in both approach and objective:

• From token merging to semantic completeness: We utilize token similarity not for merging redundant tokens, but to introduce a set coverage perspective that explicitly encourages semantic completeness among selected tokens. This represents a paradigm shift from existing methods.
• Individual saliency vs. collective semantic completeness: Current saliency-based pruning methods focus on selecting individually important tokens, which creates a critical issue: semantic incompleteness. These methods optimize for individual token importance without considering the collective semantic representation.
• Joint saliency-Coverage optimization: Our key contribution is to formulate token selection as a joint optimization over saliency and coverage.
  • Saliency: Ensures token-level importance is preserved.
  • Coverage: Guarantees selected tokens span complete semantic regions.

This dual optimization results in superior semantic completeness, which proves particularly critical for downstream reasoning tasks in MLLMs where comprehensive understanding is essential. While we share certain tools (e.g., similarity metrics) with prior works, our principal objective, and mathematical formulation represent a distinct contribution that addresses fundamental limitations in existing approaches.

Q2: It lacks a visualization of selected visual tokens for images to help us better understand how the SCOPE score-guided token pruning differs from saliency-guided token pruning. The provided visualization is not an actual image and thus is hard to interpret. For example, you should use a mask to highlight the chosen region of the image.

A2: Thank you for your thoughtful suggestion. We agree that visualizing the selected tokens directly on the original image would greatly improve the interpretability of our method.

• Key difference in token selection beavior: The fundamental distinction between saliency-guided pruning and our SCOPE lines in their selection strategies:
  • Saliency-based Pruning focuses only on the most attended tokens (typically the main object or salient featurs), which may lead to semantic redundancy or incompleteness.
  • SCOPE jointly considers both saliency and semantic coverage, aiming to select tokens that are important and semantically complete to form a more comprehensive representation.

Visualization Commitment: We have already prepared visualizations that highlight the selected regions on the input images using masks. Due to the rebuttal policy, we cannot include these visualizations in this response. We commit to incorporating these detailed visual comparisons in the camera-ready version to better illustrate the behavior of our method.

Q3: How does the proposed method compare to randomly chosen tokens in terms of performance? Random selection can also enable evenly pruned tokens, thus potentially providing more complete coverage, though this is a very simple and naive heuristic.

A3: Thank you for the valuable suggestion. The comparison between our method and random token selection is already included in the submission (Table 3), and for clarity, we extract the relevant results in Table B1 below:

Table B1: Performance comparison between Random selection and our SCOPE.

While random pruning may offer broad token coverage due to its uniformity, it fails to retain critical visual semantics. In contrast, our method jointly considers saliency (to select informative tokens) and semantic coverage, leading to stronger semantic preservation and consistently higher accuracy.

Q4: In Figure 2, the coverage for SCOPE is actually lower than for coverage only, but in the ablation study, SCOPE performs better than coverage only. Can you provide some explanation to help me understand what this means?

A4: Thank you for your question. The key insight is that maximizing coverage alone does not necessarily lead to optimal model performance.

• Coverage-only strategy indeed promotes semantic coverage by selecting a wide range of tokens, it often includes tokens with low saliency.
• Our SCOPE method formulates token selection as a joint optimization of saliency and coverage. This means that selected tokens are not only semantically complete, but also highly informative. Though this combined objective may result in a slightly lower raw coverage score compared to the coverage-only strategy, the selected tokens better represent the essential semantics of the image, leading to improved model performance.

For better understanding, we provide an example to illustrate the difference of selected tokens among different strategies. As we cannot upload figures, we will briefly describe the visualization of our token pruning strategies on an image of a cat and a banana.

1. Saliency-only: Selected tokens are mainly concentrated on the cat and banana, demonstrating object-level focus by pruning the background.
2. Coverage-only: Tokens are spread across the image, preserving global context but potentially missing important object details.
3. Our SCOPE: High token density is maintained on the cat and banana, while a sparse set of tokens is strategically kept for the background. This captures critical object features without discarding essential scene context.

We have prepared visualizations to support this observation and will include them in the final version of the paper.

Q5: In Figure 4, when the number of tokens is relatively large (for example, 48), it seems that VisionZip is starting to beat SCOPE. However, in Tables 1 and 2, when retaining 64 / 160 tokens, SCOPE is significantly better than VisionZip. This is inconsistent, as if Figure 4 is true, then VisionZip should be better when the number of retained tokens is 64. How do you explain this?

A5: Thank you for your careful observation. We have re-checked the results and confirmed their correctness. The apparent inconsistency between Figure 4 and Tables 1 and 2 mainly arises from performance fluctuations that commonly occur when the number of retained tokens is small.

In low-token regimes (e.g., 32 or 48 tokens), token pruning methods are more sensitive to which specific tokens are preserved, and small differences in selection may lead to noticeable performance variation. For example, as shown in Figure 4 on the SQA dataset, retaining 32 tokens leads to better performance than 48 tokens.

While performance may fluctuate at certain token counts, SCOPE generally achieves better results than VisionZip across a wide range of benchmarks and token budgets. We will make this clearer in the final version to ensure clarity.

Q6: The authors can provide a breakdown of their method's latency to better understand why the significant token reduction does not translate into substantial latency reduction.

A6: Thank you for the helpful suggestion. To better understand the relationship between token reduction and actual latency improvement, we provide a detailed latency breakdown based on LLaVA-NeXT 7B, shown in Table B2.

Table B2: Efficient Analysis on LLaVA-NeXT 7B. The experiments are conducted on 4*A100 on POPE.

As shown above, SCOPE significantly reduces the number of visual tokens (from 2880 to 160), resulting in a 6.2$\times$ speedup in the LLM prefilling stage, which is the most time-consuming part of the pipeline. However, this reduction does not linearly translate to overall latency reduction, primarily because:

• The Pre-LLM stage (vision encoding and token pruning) becomes more dominant after pruning, as its computation is less affected by the token count.
• The text tokens themselves also contribute non-negligible computation, and their processing cost remains constant regardless of visual token pruning.

We sincerely appreciate the reviewer’s valuable and constructive feedback. Below, we address your questions as follows:

Q1: Comparison with methods of vision token compression for ViTs.

A1: Thank you for the suggestion. We agree that established ViT-based token pruning methods are important baselines. Following your suggestion, we compare our method with a representative ViT-based token pruning approach, Token Merging (ToMe) [1], which reduces token redundancy while preserving visual feature quality. The results on LLaVA-1.5 7B are presented in Table A1.

Table A1: Results of SCOPE and ViT token pruning method.

From Table A1, we can see that our SCOPE consistently outperforms ToMe under the same token budget. These results highlight the importance of developing vision-side token pruning strategies specifically tailored for MLLMs, and validate the effectiveness of our proposed method. This trend is also observed in prior efficient MLLM designs such as VisionZip, which adopt post-ViT pruning strategies.

The key reasons behind the performance differences are:

• Different semantic requirements:
  • ViT-based pruning methods aim to retain discriminative features for classification tasks, which primarily require category-level information.
  • MLLMs demand more comprehensive and fine-grained semantic representations to support image understanding and reasoning.
• Different pruning stages:
  • ViT-based methods prune tokens at intermediate layers to reduce computation within the ViT backbone. In ViTs, as layers go deeper, the relationships among tokens evolve. Shallow layers focus primarily on local spatial structures, while deeper layers capture more global and semantic dependencies. Therefore, pruning too early may disrupt the development of critical token interactions before they are fully established.
  • Vision-side token pruning for MLLMs, in contrast, is applied after the full forward pass of the ViT. This allows pruning decisions to be based on the final visual representations, which already encode rich semantic information. Such a design better aligns with the needs of MLLMs, enabling more informed token selection based on comprehensive embeddings.

We will include this comparison and discussion in the final version to provide a more comprehensive evaluation.

We thank the reviewer again for the insightful comment. Although existing ViT-based pruning methods are not directly compatible with MLLMs, we believe it is a promising future direction to design ViT-side token merging/pruning that is also aware of multimodal semantic needs.

[1] Token merging: Your vit but faster, ICLR 2023.

Second Invitation for Further Discussion

Dear Reviewer h3Fi:

Given the review timeline and policy, we would be grateful for any indication of whether our responses have satisfactorily addressed your concerns, or if further clarification is needed.

We remain committed to improving our manuscript based on your expertise.

Thank you for your time and consideration.

Best regards,

The authors

Gentle Reminder: Reviewer Discussion Participation

Dear Reviewer h3Fi,

Thank you again for your time and effort in reviewing our submission.

Just a gentle reminder that the discussion phase will end soon. We truly appreciate your earlier review, and would be grateful for any additional comments or clarifications you might share during this phase.

As noted in the NeurIPS policy, submitting the “Mandatory Acknowledgement” does not replace participation in the discussion. Your engagement would help ensure a more thorough and balanced evaluation.

Please let us know if we can clarify any points from the rebuttal or provide further information.

Best regards,

The Authors of Submission 4718

Clarification Request Regarding Remaining Concerns

Dear Reviewer h3Fi,

Thank you again for your thoughtful review and for the time you've dedicated to evaluating our submission.

As per the NeurIPS 2025 discussion guidelines, reviewers are encouraged to communicate clearly whether their concerns have been addressed:

“If authors have resolved your questions, do tell them so.”

“If authors have not resolved your questions, do tell them so too.”

We have provided detailed responses to the concerns you previously raised. We kindly ask whether there are any remaining issues you feel have not yet been addressed.

Just a gentle reminder that the discussion phase will end soon. We would truly appreciate any additional comments you might have.

Best regards,

The Authors

The author provided the response. Could you please check if the concerns are fully addressed?

Thanks, AC