Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs

We sincerely thank the reviewer for the thorough review and valuable time. We are encouraged by the recognition of our method as novel and our performance as superior. After carefully considering the comments, we provide the following responses:

1. Evaluation benchmarks:

   We follow the most commonly adopted benchmarks used in open-source MLLMs, covering general visual question answering, document understanding and OCR, as well as video understanding. We plan to extend our evaluation to a broader range of tasks in future work.

2. Calculation of the “Rel.” metric:

   The “Rel.” metric is calculated as the average of the relative accuracies for each benchmark, rather than using a single relative average accuracy across all tasks. Thanks for pointing this out — we will clarify the calculation method of the metrics in the revised version.

3. Runtime:

   We have already described the runtime in L18 of the abstract and L57 of the introduction. Here, the reported CUDA latency refers to the actual runtime during inference. Furthermore, we provide a detailed analysis of runtime and efficiency in Section 4.6 and Table 5. Following the reviewer’s suggestion, we will include runtime latency as an additional column in Tables 1–4 in the revised version.

4. Explanation of token number $K$:

   The number of retained tokens $K$ here is not a hyperparameter but rather an input to the algorithm. The goal of our work is to improve the inference efficiency of MLLMs to meet various deployment constraints. Therefore, $K$ can be set according to the available computational budget. For example, the original LLaVA-NeXT-7B model has an inference latency of approximately 280 ms per sample. In latency-sensitive scenarios requiring < 60 ms inference time, the number of visual tokens must be reduced from 2880 to 320. In this case, we should set $K \le 320$. In short, users can flexibly set the retained token number $K$ based on their specific computational constraints, and our CDPruner is capable of adapting to different values of $K$ while consistently achieving strong performance. The values of $K$ presented in the tables are intended to illustrate performance under different reduction ratios, following previous works. The high-resolution input of 2880 visual tokens, as well as the token number in video scenarios, are fixed for fair comparison and do not directly determine the value of $K$.

5. Abbreviation typo:

   We thank the reviewer for the careful review. We will correct this typo in the revised version.

We sincerely thank the reviewer for the effort and constructive feedback. We are encouraged by the recognition of our method as novel and effective, the acknowledgment of our experiments as extensive, and the appreciation of the paper as well written and easy to understand. After thoroughly reading the comments, we provide the following point-by-point responses:

1. Computational cost of text feature extraction:

   For text feature extraction, we follow the implementation of TRIM. Since the extraction of visual and textual features is entirely independent, we allocate separate CUDA streams for each, allowing both processes to run in parallel. As text feature extraction is generally faster than visual feature extraction, this component does not introduce additional overhead in practice. The corresponding pseudocode is as follows:

   image_stream = torch.cuda.Stream()
   text_stream = torch.cuda.Stream()
   
   with torch.cuda.stream(image_stream):
       image_features = self.vision_tower(images)
   
   with torch.cuda.stream(text_stream):
       text_inputs = self.text_tokenizer(texts)
       text_features = self.text_tower(**text_inputs)
   
   torch.cuda.synchronize()
   

2. Compatibility with token pruning methods within LLM:

   Our proposed method is fully compatible with any pruning method within LLMs. To demonstrate this, we combine CDPruner with three representative in-LLM pruning methods: FastV, PDrop, and SparseVLM. To retain $m$ visual tokens, we first apply CDPruner before the LLM to reduce the token number to $2m$, and then apply the in-LLM pruning method to further prune it down to $m$ tokens. The experimental results are as follows:

   Method	VQAV2	GQA	VizWiz	SQA-IMG	TextVQA	POPE	MME	MMB	MMB-CN	MM-Vet	Acc.	Rel.
   Upper Bound, All 576 Tokens (100%)
   LLaVA-1.5-7B	78.5	61.9	50.1	69.5	58.2	85.9	1506.5	64.7	58.1	31.3	63.4	100.0%
   Retain 128 Tokens (-77.8%)
   CDPruner	76.6	59.9	52.8	69.0	56.2	87.7	1431.4	63.1	55.0	32.8	62.5	99.0%
   CDPruner+FastV	76.4	58.2	51.3	68.3	57.4	81.8	1469.6	62.8	56.3	32.3	61.8	98.1%
   CDPruner+PDrop	76.9	59.6	50.9	67.9	57.4	85.2	1473.9	63.6	55.9	32.3	62.3	98.7%
   CDPruner+SparseVLM	76.8	59.5	51.3	67.7	57.4	86.1	1440.4	63.1	56.3	32.6	62.3	98.7%
   Retain 64 Tokens (-88.9%)
   CDPruner	75.4	58.6	53.4	68.1	55.3	87.5	1415.1	61.1	53.2	30.5	61.4	97.0%
   CDPruner+FastV	75.3	57.4	52.6	69.6	56.6	82.2	1417.1	62.4	53.1	30.3	61.0	96.5%
   CDPruner+PDrop	74.3	57.9	52.3	69.9	56.1	84.9	1406.6	62.2	54.2	30.0	61.2	96.7%
   CDPruner+SparseVLM	75.7	58.3	52.4	69.5	55.9	86.3	1434.4	61.8	54.3	30.6	61.7	97.4%
   Retain 32 Tokens (-94.4%)
   CDPruner	73.6	57.0	53.1	69.5	53.2	87.9	1373.0	59.6	49.6	27.8	60.0	94.3%
   CDPruner+FastV	73.4	56.1	52.9	68.7	54.8	81.1	1368.9	60.0	50.9	28.3	59.5	93.9%
   CDPruner+PDrop	73.5	56.5	52.5	69.8	54.4	83.2	1366.1	60.1	51.2	28.9	59.8	94.4%
   CDPruner+SparseVLM	74.1	56.9	52.8	69.3	54.5	85.5	1392.7	60.1	52.4	29.3	60.5	95.4%

   These combined methods yield further performance improvements over the original CDPruner, clearly demonstrating that our approach is fully compatible with various in-LLM pruning approaches.

In summary, through the implementation of dual CUDA streams, CDPruner does not introduce additional computational cost for text feature extraction, confirming its efficiency. Meanwhile, experimental results show that CDPruner is also compatible with a wide range of pruning methods within LLMs and achieve improvements, fully demonstrating its compatibility. We hope the above response addresses your concerns.

We thank the reviewer for the detailed review and valuable feedback. We are encouraged by the recognition of our method as novel and offering a unique perspective, as well as the acknowledgment that our evaluation is comprehensive and thorough. After careful consideration of the comments, we provide the following point-by-point responses:

1. Additional embedding models for instruction relevance:

   To compute the relevance between visual tokens and user instructions, it is necessary to obtain visual and textual embeddings of the same dimension and ensure they are situated in a unified semantic space. Therefore, the most suitable choice is a model that has been pre-trained through language-image contrastive pretraining. If the visual features in the MLLM are already well aligned with text, an additional embedding model may not be required. Here, we introduce the CLIP model for Qwen2.5-VL to validate the effectiveness of an additional embedding model. The corresponding experimental results are as follows:

   Method	AI2D	HallBench	MMB-EN	MMB-CN	Acc.	Rel.
   Upper Bound, All 1296 Tokens (100%)
   Qwen2.5-VL-7B	84.4	46.8	83.9	83.4	74.6	100.0%
   Retain 512 Tokens (-60.5%)
   CDPruner	82.9	42.5	82.2	82.6	72.6	96.5%
   CDPruner+CLIP	83.7	43.6	83.1	82.1	73.1	97.5%
   Retain 256 Tokens (-80.2%)
   CDPruner	80.5	40.1	80.9	79.9	70.4	93.3%
   CDPruner+CLIP	81.7	41.5	82.6	81.1	71.7	95.3%
   Retain 128 Tokens (-90.1%)
   CDPruner	76.0	37.2	76.2	76.5	66.5	88.0%
   CDPruner+CLIP	79.3	38.1	81.3	78.1	69.2	91.5%

   After incorporating CLIP into Qwen2.5-VL, CDPruner consistently achieves improved performance across various benchmarks. This result indicates that accurate instruction relevance estimation plays a crucial role in effective token pruning and provides new insights for adapting CDPruner to different MLLM architectures. We will continue to explore the value of additional embedding models for token pruning in our future work.

2. Extension to multi-turn dialogues:

   In multi-turn scenarios, it is possible to dynamically adjust the pruned tokens based on the current instruction by caching the key-value pairs of all visual tokens. Specifically, during the prefill stage, instead of discarding all pruned tokens, we can encode and store their key and value representations. In subsequent dialogue turns, we can then retrieve relevant tokens from the cache based on the current instruction, enabling dynamic token adjustment. Following the reviewer’s suggestion, we directly apply CDPruner to the MMDU benchmark [1] without performing dynamic token adjustment to evaluate the effect of pruning under this challenging setting:

   Method	C	R	VP	LC	AA	IRU	Acc.	Rel.
   Upper Bound, All 576 Tokens (100%)
   LLaVA-1.5-7B	34.8	32.7	39.4	65.3	47.4	39.5	42.9	100.0%
   Retain 128 Tokens (-77.8%)
   TRIM	35.7	34.2	38.7	64.6	46.8	39.2	42.8	99.8%
   CDPruner	36.2	34.9	40.0	66.2	48.0	40.8	44.0	102.6%
   Retain 64 Tokens (-88.9%)
   TRIM	35.6	34.1	37.1	63.8	44.8	37.7	41.7	97.2%
   CDPruner	36.1	34.4	38.6	64.5	46.2	39.0	42.8	99.8%
   Retain 32 Tokens (-94.4%)
   TRIM	35.4	34.0	36.1	62.8	44.2	36.9	41.2	96.0%
   CDPruner	35.6	34.0	36.7	62.9	44.6	38.0	41.5	96.7%

   TRIM prunes tokens based on their relevance to the instruction, which is suboptimal for multi-turn dialogue. If the subsequent question differs significantly from the previous one, the retained tokens may no longer be relevant, resulting in degraded performance. In contrast, CDPruner incorporates diversity modeling via DPP, which enables it to preserve more informative and comprehensive visual content while still considering relevance. As a result, it maintains better performance even in multi-turn scenarios. Experimental results show that CDPruner consistently outperforms TRIM under different reduction ratios, demonstrating its adaptability to multi-turn dialogues.

3. In-depth analysis of CDPruner’s effectiveness:

   This work identifies visual diversity and instruction relevance as two key factors for token pruning in MLLMs. Previous methods typically considered only one of these factors. Methods focusing solely on diversity (e.g., DART, DivPrune) tend to retain too many tokens irrelevant to the current question, while a large number of tokens in key areas are discarded, leading to loss of critical information. On the other hand, methods focusing only on relevance (e.g., FastV, TRIM) often retain redundant tokens while discarding informative ones from low-relevance regions, also resulting in performance degradation. Naively combining both factors in a two-stage pruning strategy does not effectively address this issue, as each stage may still retain unimportant tokens. In contrast, CDPruner jointly models both relevance and diversity using DPP, and selects the optimal subset from a global perspective. This enables it to retain the most relevant tokens while removing redundancy, achieving efficient pruning without sacrificing performance.

4. Potential challenges across different architectures and corresponding strategies:

   Since CDPruner performs token pruning before the language model, the visual encoder is more crucial to our method. For MLLMs whose visual encoders are paired with a corresponding text encoder (e.g., LLaVA-1.5, LLaVA-OneVision), we directly compute cosine similarity between visual and textual embeddings for relevance estimation. For models without a paired text encoder (e.g., Qwen2.5-VL, InternVL3), alternative strategies can be adopted, such as using embeddings from the language model’s embedding layer to compute similarity, leveraging text-visual attention from shallow layers of the language model, or introducing an additional embedding model (e.g., CLIP) to assist relevance computation. Designing a general pruning strategy for diverse MLLM architectures remains a promising direction, and we will continue to explore it in future work.

5. Font size issue:

   Thanks for pointing this out. We will increase the font size in the corresponding figures and tables in the revised version to improve the readability of the paper when printed.

We sincerely appreciate the reviewer’s recognition of our work. And we hope that the above analysis of CDPruner’s effectiveness and the discussion on adaptation strategies to various model architectures have addressed your concerns and can offer valuable insights for future research in this field.

[1] Liu, Ziyu, et al. "Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms." Advances in Neural Information Processing Systems 37 (2024): 8698-8733.

We sincerely thank the reviewer for the time and effort in evaluating our work. We are encouraged by the recognition of our method as elegant and our experiments as comprehensive and impressive. After carefully reading the comments, we would like to address the reviewer’s concerns regarding the computational overhead of DPP in our CDPruner as follows:

• Computational overhead with respect to the initial token number $n$:

  We fix the number of kept tokens to 64 and measure the runtime of a single DPP step (averaged over 10,000 runs) as a function of the initial token number $n$:

  Initial Token Number ($n$)	144	288	576	1152	2304
  Latency (ms)	0.86	1.09	1.37	2.00	3.64

  The results show that the overall computational overhead does not increase significantly with the initial token number $n$. Even when the number of visual tokens exceeds 2k, the latency of a single run remains below 4ms, which is well within an acceptable range for practical applications.

• Computational overhead with respect to the kept token number $m$:

  We fixed the number of initial tokens to 576 and measured the runtime of a single DPP step (averaged over 10,000 runs) as a function of the kept token number $m$:

  Kept Token Number ($m$)	16	32	64	128	256
  Latency (ms)	0.45	0.77	1.37	2.73	5.84

  The results show that while the runtime increases more noticeably with $m$ compared to $n$, the latency per run still remains very low. In practical scenarios, the number of kept tokens $m$ is typically much smaller than the initial number $n$, so the overall DPP runtime remains negligible and does not affect the real-time application of the algorithm.

• Further optimization of algorithmic complexity:

  Our current implementation adopts fast greedy MAP inference for DPP, with a theoretical complexity of $\mathcal{O}(nm^2)$. However, for MLLMs with dynamic high-resolution inputs (e.g., LLaVA-NeXT, InternVL3), the input image is first divided into equally sized patches, and the DPP can be performed in parallel across different patches. This significantly reduces the computational complexity: for an image with $k$ patches, the global complexity $\mathcal{O}((kn)(km)^2) = \mathcal{O}(k^3nm^2)$ can be reduced to a local complexity of $\mathcal{O}(knm^2)$.

  Additionally, by applying a sliding window of width $w (w < m)$ during DPP computation, the complexity can be further reduced to $\mathcal{O}(wnm)$ [1]. Furthermore, incorporating a Markov chain approximation into the DPP sampling can reduce the complexity to the ideal $\mathcal{O}(m^3)$ [2].

• Further reduction of runtime latency:

  Given that the overall computing process of DPP is sequential, in real-time applications with extremely strict latency requirements, we can further optimize inference latency by implementing custom operators, enabling low-level and system-level acceleration.

In summary, we believe the computational overhead of DPP does not hinder the real-time applicability of CDPruner. Moreover, various strategies exist to further reduce the inference latency under different deployment scenarios. We hope the above responses effectively address your concerns, and we sincerely hope you would consider increasing the rating if your concerns have been resolved.

[1] Chen, Laming, Guoxin Zhang, and Eric Zhou. "Fast greedy map inference for determinantal point process to improve recommendation diversity." Advances in neural information processing systems 31 (2018).

[2] Kang, Byungkon. "Fast determinantal point process sampling with application to clustering." Advances in Neural Information Processing Systems 26 (2013).