Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Covering

Sincerely appreciate your responsible work and constructive questions! We provide a point-to-point feedback on the weakness (W) and question (Q)

W1, Q1: Automatic Estimation of $K_{\rm p}$: Currently, MoB requires manual selection of $K_{\rm p}$. Could the authors consider estimating $K_{\rm p}$ from prompt length or coupling strength $\eta$? Would meta-learning or lightweight heuristic predictors help automate this?

A: Thank you for this forward-thinking question, which is highly align with our ongoing research. In fact, deriving a analytical expression for the optimal $K_{\rm p}$ from prompt length $L$ and coupling strength $\eta$ is highly challenging at present. Therefore, to make the automatic estimation of $K_{\rm p}$ more tractable, we have developed an elegant and practical framework in our follow-up work.

Instead of solving a precise $K_{\rm p}$ value, this framework learns to adaptively classify the coupling pattern of incoming samples and applies a pre-configured budget configuration $K_{\rm p}$. It operates as follows:

1. During inference, the framework maintains a running statistic of an approximated $\hat{\eta}$ for the processed samples. We observe that as more samples are processed, this statistic naturally reveals the same bimodal distribution pattern identified in our paper (Figs. 1b & 9).
2. From this observed distribution, the framework dynamically determines a classification threshold that separates the two patterns.
3. Each new sample is then classified into 'weak' or 'strong' coupling using this live, data-driven threshold, and the corresponding optimal $K_{\rm p}$ is applied.

This approach directly leverages the coupling strength $\eta$, as you suggested, in a more practicable way. It can be viewed as a form of lightweight, online meta-learning that adapts to the specific data distribution encountered during deployment.

W1, Q2: Computational Cost of Hausdorff Distance: While theoretically elegant, the computation of Hausdorff distance in high-dimensional spaces can be expensive. Have the authors considered approximations or alternatives that maintain performance?

A2: Thank you for this very important and forward-thinking question. We agree that the computational cost of the exact Hausdorff distance is a critical consideration. We address this with a quantitative analysis of the practical costs, followed by a discussion of efficient alternatives.

Quantitative Analysis: We provide a detailed cost breakdown for LLaVA-1.5-7B and LLaVA-Next-7B below:

Computational cost (TFLOPs) in LLaVA-1.5-7B, where $N=576, L=10, d=4096$.

Computational cost (TFLOPs) in LLaVA-Next-7B, where $N=2880, L=10, d=4096$.

As the tables clearly show, while the complexity of computing Hausdorff distance, i.e., $O(NLd)$, is intuitively heavy in high-dimensional spaces, it remains orders of magnitude smaller than both the MoB itself and the model's forward propagation. Thereby, it cannot be a practical bottleneck of MoB. Considering your concern, we also present two powerful approximation techniques:

• Heuristic Sampling: It computes the distance on smaller support sets of the tokens ($\mathcal{V}'\subset \mathcal{V},\mathcal{P}'\subset\mathcal{P}$), which can be constructed via random sampling [1] or more advanced heuristics like Key-Norm selection [2, 3]. This reduces complexity to $O(N'L'd)$.
• Random Projections: For a more theoretically-grounded approach, the Johnson-Lindenstrauss (JL) lemma [4] allows us to project embeddings to a much lower dimension ($d'\ll d$) while preserving geometric structure, reducing complexity to $O(NLd')$.

In summary, the cost of computing Hausdorff distance is already negligible in practice, as shown by our quantitative analysis, and can be further optimized via well-established approximation methods if needed. This ensures the broad applicability and efficiency of our framework.

References

[1] Zichen Wen, et al. Stop looking for important tokens in multimodal language models: Duplication matters more. preprint, 2025.

[2] Akhauri Y, AbouElhamayed A F, Gao Y, et al. Tokenbutler: Token importance is predictable. preprint, 2025.

[3] Guo Z, et al. Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters. CoRR, 2024.

[4] Larsen K G, Nelson J. The Johnson-Lindenstrauss Lemma Is Optimal for Linear Dimensionality Reduction. LIPIcs, 2016.

W2: Assumption Validity: The theoretical guarantees depend on the assumption of Lipschitz continuity of the model in the Hausdorff metric space. While reasonable in practice, this may not hold universally.

A: Thanks for your constructive concern; we justify Assumption 1 in three aspects:

Theoretical Grounding: Prior work [1, 2] has shown that Transformers are approximately Lipschitz continuous w.r.t. standard matrix norms, e.g., the Frobenius norm, under the finite and compact input conditions we consider (Line 98). A standard inequality relates the Frobenius norm to the Hausdorff distance: $\frac{1}{\sqrt N}\Vert\mathcal X\_1-\mathcal X\_2\Vert_{\_F}\le d\_H(\mathcal X\_1,\mathcal X\_2)\le\Vert\mathcal X\_1-\mathcal X\_2\Vert_{\_F}$. This implies that any model Lipschitz continuous w.r.t. the Frobenius norm is also Lipschitz continuous w.r.t. the Hausdorff distance.

Empirical Evidence: Vignac et al. [3] have measured the empirical local Lipschitz constant of Transformers on real-world data and found it to be 100-1000x smaller than the theoretical worst-case bound. This indicates that practical models already lie in a well-behaved, near-Lipschitz regime. Besides, Kim et al. [4] have shown that the successful training of deep Transformers relies on controlling this constant. Architectures without such implicit or explicit constraints often fail to converge, suggesting that the well-trained VLLMs most likely adhere to a finite Lipschitz constant.

Consistency with Prior Work: We wish to clarify that this assumption is not novel to our work but is a common theoretical premise in the visual token pruning community, having been adopted by prior methods such as DivPrune [5] and DART [6]. Our work builds upon this established foundation to derive deeper insights.

References

[1] Dasoulas G, et al. Lipschitz normalization for self-attention layers with application to graph neural networks. ICML, 2021.

[2] Yudin N, et al. Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers. preprint, 2025.

[3] Vignac C,  et al. Understanding the regularity of self-attention with optimal transport. preprint, 2023.

[4] Kim H, et al. The lipschitz constant of self-attention. ICML, 2021.

[5] Alvar S R, et al. Divprune: Diversity-based visual token pruning for large multimodal models. CVPR, 2025.

[6] Wen Z, et al. Stop looking for important tokens in multimodal language models: Duplication matters more. preprint, 2025.

W3, Q3: Performance with Fine-tuning: As MoB is training-free, have the authors tested its combination with lightweight fine-tuning post-pruning? Would such a setup improve performance under aggressive compression?

A: Thank you for this insightful suggestion. While our work focused on the training-free aspect, we agree this is a promising direction. A key advantage of MoB is that it provides a principled, informative sparse input to the LLM. A subsequent lightweight fine-tuning (e.g., LoRA) would then allow the model, originally trained on dense data, to adapt its attention mechanisms to this new distribution. We hypothesize this would significantly improve performance under aggressive compression and have added it as a key direction for future work.

Q4: Applicability to Language Token Pruning: While MoB focuses on visual tokens, could the same methodology be adapted to prune language tokens, especially for long-context LLMs?

A: Thank you for this constructive question. While naively applying MoB on language tokens might not yield the same degree of success, our theoretical framework has the fundamental generalizability for the language domain.

In the language domain, we can create a powerful analogy:

• Visual Preservation (VP) is equivalent to preserving the broader context of long language representations.
• Prompt Alignment (PA) is equivalent to extracting the golden evidence—the most critical sentences directly relevant to the facts.

The trade-off between VP and PA, a key insight of our theory, is therefore fundamentally the trade-off between Context and Golden Evidence. We believe this principle has significant implications for several typical applications in long-text LLMs:

• RAG: Balancing recall (enough Context to reason) vs. precision (the Golden Evidence that answers the query).
• Summarization & LLM Memory: Balancing coherence (the Context/flow) vs. conciseness (the Golden Evidence/key facts).

Our framework provides a novel, principled view for analyzing and optimizing these fundamental trade-offs, highlighting its significance for the society of long-context LLMs.

Thanks for your comments. To address your concerns, we provide our point-to-point clarification as follows

W1 & Q1: Although a new method is proposed, the performance improvement of MoB is relatively modest in some experimental settings, which may fall short of expectations in certain scenarios.

A1: Thank you for your feedback. We respectfully argue that the performance improvement of MoB is not modest, and we would like to highlight two key aspects of our contribution: its significant and robust empirical performance and its foundational theoretical insights.

1. Significant and Robust Empirical Performance: We believe the term "modest" may overlook two critical patterns in our results.

• Consistent Superiority: Across all three tested model families (LLaVA-1.5, LLaVA-Next, Qwen2-VL, and Video-LLaVA) and all pruning rates, MoB consistently outperforms the strongest competing baselines in average performance. For example, MoB achieves average performance gains of 2.7% (LLaVA-1.5), 1.5% (LLaVA-Next), and 1.2% (Qwen2-VL) in the image domain and achieves a 97.9% average performance at a very aggressive 93.4% token reduction rate, surpassing the next-best method by 1.6% in the video domain.
• Widening Lead Under Aggressive Pruning: Crucially, MoB's performance advantage becomes more pronounced under more challenging, high-ratio pruning scenarios. On LLaVA-1.5-7B, for instance, our lead over the strongest baseline grows from +1.4% at a 77.8% reduction rate to a more substantial +2.7% at an 88.9% reduction rate. This demonstrates the superior robustness of our method when efficiency demands are highest.

We contend that this consistent and robust outperformance, especially under aggressive compression, is a significant achievement.

2. Foundational Theoretical Contribution: Beyond the performance numbers, we encourage the reviewer to consider the value of our theoretical contributions, which we believe are a core part of our work.

• Provable Guarantees: Unlike most heuristic-based pruning methods, MoB is grounded in a formal theoretical framework that provides a provable performance bound (Theorem 2). This ensures its reliability.
• Novel Insights for the Community: More importantly, this paper presents the first in-depth analysis of the key factors in visual token pruning—Visual Preservation (VP), Prompt Alignment (PA), prompt-visual coupling ($\eta$), and budget ($K$)—and formalizes their interplay. MoB is a direct and effective application of these insights. We believe these foundational theories will motivate the community and inspire more powerful, principled pruning methods in the future, representing a profound potential impact.

W2 & Q2: The interpretation of some experimental results could be more in-depth. The paper does not fully explore the applicability and limitations of different pruning strategies.

A2: Thank you for your comment. We respectfully clarify that a central motivation and contribution of our work is precisely to provide an in-depth analysis of the applicability and limitations of different pruning strategies, and how to incorporate them to derive a robust visual token pruning method. We have demonstrated this extensively in our manuscript from theoretical, empirical, and ablation perspectives.

1. Theoretical Foundation: Our key theoretical insight is that the effectiveness of a pruning strategy is contingent on the prompt-visual coupling pattern (η). We explicitly analyze this in our discussions of Lemma 2 (Lines 143-147) and Theorem 2 (Lines 232-234). In these sections, we formally reveal that Prompt Alignment (PA) is crucial for weak-coupling tasks, while Visual Preservation (VP) is favored under strong-coupling conditions.

2. Empirical Validation in Main Results: This theoretical prediction is directly validated by the trends in our main experimental results in Tables 1 and 4. As we explicitly state in our analysis in Lines 252-255: "...single-objective baselines exhibit complementary strengths under different coupling patterns...". This discussion confirms that PA-based methods excel on weak-coupling benchmarks, while VP-based methods perform better on strong-coupling ones.

3. Targeted Ablation Studies: Furthermore, we conducted a targeted ablation study (Lines 284-299), the results are presented in Figure 4, to isolate and verify this exact relationship. By systematically varying the budget allocation $K_{\rm p}$ (considered as the quality of prompt alignment), we empirically confirmed that PA is more critical for weak-coupling tasks and VP for strong-coupling ones, thus validating our core theoretical premise.

In summary, we believe our manuscript provides a comprehensive, multi-faceted analysis of the applicability and limitations of different pruning strategies. We are confident that these detailed discussions, located throughout the paper, fully address your concern.

W3 & Q3: While the method shows improvements, compared to existing approaches such as CrossGET, LLaVA-PruMerge, TokenPacker, Turbo, and AIM, which also demonstrate similar effectiveness, the paper lacks comparison and discussion with these methods.

A3: Thank you for your suggestion to compare with a broader set of baselines. We would first like to clarify that our original manuscript already contains a strong suite of 13 baselines (Appx. C.3), including the mentioned LLaVA-PruMerge [2], where MoB consistently demonstrates state-of-the-art performance (Tables 1-4). Thank you for suggesting these excellent works. To further address your concern, we have conducted additional experiments to provide a direct comparison against CrossGET [1] and AIM [5]. The comparison results are presented below:

As the new results clearly demonstrate, MoB’s effectiveness is not "similar" but substantially superior. MoB achieves a 5.4% average performance gain over the CrossGET under a ~31% FLOPs reduction, while At a ~73% FLOPs reduction, MoB achieves a 4.1% and 10.6% average performance gain over the AIM and LLaVA-PruMerge under a ~73% FLOPs reduction. As for a more aggressive reduction, e.g., 88%, MoB still achieve 6.8% and 9.0% average performance gain over the AIM and LLaVA-PruMerge. We believe these significant margins validate the effectiveness of our approach.

We did not include Turbo [4] and TokenPacker [3] for the following principled reasons:

• The repository of Turbo's code is still invalid, and its does not report results on any of the standard benchmarks used by the community, making a fair and reproducible comparison impracticable.
• Although TokenPacker is also designed for acceleration of VLLM, it belongs to a different research category. It is a novel efficient VLM instead of a visual token pruning method. TokenPacker requires extensive training with extra data, not only pre-training (1.2M data) but also STF (1.5M data). Comparing it directly with a training-free, inference-time visual token pruning method like MoB would be unfair. Crucially, these approaches are orthogonal; a pruning method like MoB could potentially be applied on top of an efficient VLM like TokenPacker for even more aggressive acceleration.

Finally, we will add these new comparisons and a detailed discussion to the revised manuscript to provide readers with a more comprehensive understanding of MoB's contribution. We hope this response alleviates your concerns regarding MoB's performance.

[1] Shi D, et al. Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers. preprint, 2023.

[2] Shang Y, et al. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. preprint, 2024.

[3] Li W, et al. Tokenpacker: Efficient visual projector for multimodal llm. IJCV, 2025.

[4] Ju C, et al. Turbo: Informativity-driven acceleration plug-in for vision-language large models. ECCV, 2024.

[5] Zhong Y, et al. Aim: Adaptive inference of multi-modal llms via token merging and pruning. ICCV, 2025.

We sincerely appreciate your responsible, careful work and insightful questions! We have corrected the format problem of Fig. 3 and provide a point-to-point feedback as follows

W2 & Q1: $\eta$ is not directly observable in real-world settings. How can $\eta$ be inferred when the task type is unknown, such as in open-domain or user-driven inference? Does MoB require pre-defined task labels or dataset-specific heuristics to function, and how does this constrain its applicability in deployment?

A1: Thank you for this critical question about real-world applicability. We provide an explanation in two aspects:

A Practical Strategy for Open-Domain Inference: MoB does not require pre-defined "task type" labels. Instead, it can perform online classification of a sample's coupling pattern. The practical strategy is a two-stage process:

• Offline Calibration: For a given target model, we first analyze the η distributions on a set of representative benchmarks to establish a robust classification threshold that distinguishes between weak and strong coupling patterns.
• Online Classification: For each incoming query, we perform an online computation of its Hausdorff distance (Alg. 2, Appx. B, with a tractable bilinear complexity of $O(NLd)$). The sample is then classified based on this value and the pre-calibrated threshold, and the corresponding $K_{\rm p}$ is applied.

We acknowledge the online computation introduces a manageable cost, but it does not fundamentally alter MoB's superior performance-efficiency trade-off.

An Advanced, Fully-Adaptive Framework: Furthermore, we have explored this direction deeply in our subsequent work and developed a more elegant, fully-adaptive framework. It operates as follows:

• During inference, it maintains a running statistic of an approximated $\hat{\eta}$ (e.g., from shallow-layer tokens) for the samples it has processed.
• We observe that as more samples are seen, this statistic naturally reveals the same bimodal distribution pattern identified in our paper (Fig. 1b & 9).
• The framework then uses this emerging distribution to dynamically adjust its own classification threshold for subsequent samples.

The theoretical framework presented in this paper is the essential foundation that enables and validates such future advancements.

W2 & Q2: The authors assume $\eta$ is known a priori based on dataset labels, but this introduces additional supervision and cost that methods like FastV or SparseVLM do not require. Can the authors justify this additional cost, and is it fair to compare MoB against zero-cost or training-free baselines under this assumption? Furthermore, could $\eta$ be efficiently approximated via lightweight dataset sampling (e.g., estimating $\eta$ from a few examples), and if so, how robust would such estimation be across domains?

A2: Considering your concern, we offer the following points in response:

Justification of Experimental Settings: We first argue that prior methods did not intentionally avoid using η, but were fundamentally unaware of the critical role of prompt-visual coupling. A core contribution of our paper is to first identify and formalize this factor. To experimentally validate our theory, it was necessary to introduce the benchmark's η pattern as a prior.

We acknowledge that this could potentially leverage information unavailable to other methods. To ensure a fair comparison, we did not delicately fine-tune $K\_{\rm p}$ for each of the 14 benchmarks. Instead, we use a coarse-grained approach: we classify benchmarks into one of two general coupling patterns and apply one of two corresponding strategies. We believe this setup is sufficiently fair and practical for real-world applications.

$\eta$-Agnostic Evaluation: In this experiment, we apply this same budget configuration to all benchmarks, regardless of their coupling pattern.

Table 1 Comparative experiments on image understanding with the LLaVA1.5-7B under static $K_{\rm p}$

As the results demonstrate, even without the $\eta$-based prior, MoB consistently outperforms all baselines in most cases. This highlights the intrinsic advantage of our approach, independent of the $\eta$-based situation, and further strengthens the experimental results presented in our manuscript.

On Lightweight Estimation: While sampling a few examples is more lightweight, it still uses prior information from the test set and risks sampling bias. In our subsequent work, we have developed a more elegant and fair solution, which maintains a statistic of an approximated $\hat{\eta}$ for the samples it has processed and adaptively identifies the coupling pattern of the following inputs, i.e., $K_{\rm p}$, based on the statistic.

W2 & Q3: Does η need to be re-estimated or recalibrated when switching to a different model? How transferable is the chosen $K_{\rm p}$ configuration across models, and to what extent does MoB’s pruning effectiveness rely on model-specific representations?

A3: We want to clarify the absolute numerical values of $\eta$ can vary with model representations; however, it does not significantly affect the optimal $K_{\rm p}$ configuration, which remains highly transferable. We justify this from both a theoretical and an empirical standpoint:

Theoretical Rationale: As stated in our manuscript (Lines 204-205), we apply L2 normalization to all token embeddings before pruning. Thus, MoB operates in a normalized space, where all tokens are projected onto a unit sphere, meaning our pruning decisions are based on the relative angular structure of the tokens, not their absolute Euclidean distances or magnitudes, which can vary across models.

Empirical Validation: In our settings, the $K_{\rm p}$ configuration is not tuned to the specific absolute numerical value of $\eta$ for a benchmark. Instead, it is set based on the coupling pattern—that is, the relative difference between $\eta$ distributions for different task types within a single model's embedding space. This is why we were able to apply the exact same $K_{\rm p}$ configurations across four distinct VLLMs: LLaVA-1.5-7B, LLaVA-Next-7B, Qwen2-VL-7B and Video-LLaVA-7B. The results (Tables 1-4) show this fixed configuration is highly effective for all models. For instance, at an 88.9% reduction rate, MoB achieved average performance gains of 2.7% (LLaVA-1.5), 1.5% (LLaVA-Next), and 1.2% (Qwen2-VL) over the strongest baselines.

This provides powerful empirical evidence that the configuration of $K_{\rm p}$ is robust across models, and MoB's effectiveness stems primarily from these universal, task-driven geometric principles rather than relying heavily on model-specific representations.

W1 & Q4: Is MoB's framework theoretically complete? Could there exist tokens whose value arises from higher-order interactions, global reasoning, or latent semantics not directly reflected in either VP or PA distances?

A4: Thank you for this profound question that addresses the core of our work. We agree that the theoretical completeness is a critical aspect. Our response is threefold:

Formulation Setting: We clarify that higher-order interactions may influence multi-stage pruning pipelines, which prune tokens across multiple layers simultaneously. To keep the problem tractable and theoretically rigorous, our analysis and methodology explicitly focus on the single-layer pruning setting (L. 94–111), where exclusively exist the first-order interactions of input tokens and the pruned counterpart. Within this well-defined scope, we argue that Visual Preservation (VP) and Prompt Alignment (PA) are sufficient to rigorously characterize the performance bounds of a pruning algorithm.

The Advantage of a Tractable Framework: The challenge of balancing objectives in multi-stage methods, e.g., SparseVLM and MustDrop, highlights a key advantage of MoB's design. By focusing on a single, decisive pruning stage, MoB makes the pruning problem practicable and analyzable. This allows for an efficient and principled balance between VP and PA, thereby enabling us to derive a strict performance guarantee for MoB, as presented in Theorem 2.

Empirical evidence: From an experiential perspective, our solid experimental results validate the correctness of our theoretical analysis. By solely considering the trade-off between these two objectives, MoB consistently and significantly outperforms a comprehensive suite of both single-stage and multi-stage methods across various benchmarks (Table 1, 4). We therefore believe this provides powerful evidence that VP and PA are the decisive factors for visual token pruning.

We sincerely acknowledge your responsible work and constructive comments, we provide our point-to-point feedbacks as follows

W1 & Q1: Could the authors provide more intuitive explanations or visual illustrations—particularly for the key theoretical insights in Section 3.2 ?

A1: Following your suggestion, we offer a more direct explanation of the theoretical insights in Section 3.2 here and will incorporate a new visual illustration and this discussion in the appendix of the revised manuscript.

Our starting point is that optimal pruning must balance two fundamental objectives: Visual Preservation (VP) and Prompt Alignment (PA), which are separately responsible for preserving the global context $\mathcal{S}\_{\rm v}$ and the local evidence $\mathcal{S}\_{\rm p}$. Just as humans need both context and specific evidence to reason accurately, a VLM without global context (VP) risks hallucination, while one without local evidence (PA) cannot answer the specific question.

Section 3.2 reformulates token selection as a geometric covering problem, thereby establishing a relationship between the preservation quality of global context and the local evidence (measured by Hausdorff distance). Finally, Theorem 1 provides a principle for "how much effort" is separately required for pruning methods to preserve global context and local evidence with a fixed budget $K$ under a specific task type $\eta$.

W1 & Q2: Has a notation table or summary been considered to address the overloaded variables like "X", "S", "K", and “η”?

A2: Considering your constructive suggestion, we have compiled a notation table and supplied it to Appendix E of the revised manuscript to increase the readability and mathematical clarity of our manuscript. We believe this revision is beneficial to make readers better understand our works.

W2 & Q3: What empirical evidence supports Assumption 1? Would it be possible to validate this assumption through ablation or probing experiments?

A3: We justify Assumption 1 in three aspects:

Theoretical Grounding: Prior work [1, 2] has shown that Transformers are approximately Lipschitz continuous w.r.t. standard matrix norms, e.g., the Frobenius norm, under the finite and compact input conditions we consider (Line 98). A standard inequality relates the Frobenius norm to the Hausdorff distance: $\frac{1}{\sqrt N}\Vert\mathcal X\_1-\mathcal X\_2\Vert_{\_F}\le d\_H(\mathcal X\_1,\mathcal X\_2)\le\Vert\mathcal X\_1-\mathcal X\_2\Vert_{\_F}$. This implies that any model Lipschitz continuous w.r.t. the Frobenius norm is also Lipschitz continuous w.r.t. the Hausdorff distance.

Empirical Evidence: Vignac et al. [3] have measured the empirical local Lipschitz constant of Transformers on real-world data and found it to be 100-1000x smaller than the theoretical worst-case bound. This indicates that practical models already lie in a well-behaved, near-Lipschitz regime. Besides, Kim et al. [4] have shown that the successful training of deep Transformers relies on controlling this constant. Architectures without such implicit or explicit constraints often fail to converge, suggesting that the well-trained VLLMs most likely adhere to a finite Lipschitz constant.

Consistency with Prior Work: We wish to clarify that this assumption is not novel to our work but is a common theoretical premise in the visual token pruning community, having been adopted by prior methods such as DivPrune [5] and DART [6]. Our work builds upon this established foundation to derive deeper insights.

Regarding probing experiments: We agree this is a valid consideration. However, computing the global Lipschitz constant is computationally challenging. The most feasible alternative, measuring local constants, has already been thoroughly executed by Vignac et al. [3, 4]. Their work provides strong empirical validation for Assumption 1, and we believe further experiments would likely reinforce their conclusion without fundamentally altering our analysis.

References

[1] Dasoulas G, et al. Lipschitz normalization for self-attention layers with application to graph neural networks. ICML, 2021.

[2] Yudin N, et al. Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers. preprint, 2025.

[3] Vignac C,  et al. Understanding the regularity of self-attention with optimal transport. preprint, 2023.

[4] Kim H, et al. The lipschitz constant of self-attention. ICML, 2021.

[5] Alvar S R, et al. Divprune: Diversity-based visual token pruning for large multimodal models. CVPR, 2025.

[6] Wen Z, et al. Stop looking for important tokens in multimodal language models: Duplication matters more. preprint, 2025.

W3 & Q4: The experiments omit widely used benchmarks such as COCO and full VQAv2. Is there a reason for this exclusion, and could results on these datasets be included to better demonstrate generalizability?

A4: We want to explain that we have evaluated MoB in ScienceQA (SQA) (Appx. C.1), while the exclusion of COCO and the full VQAv2 benchmark is a deliberate choice based on their technical limitations for visual token pruning. We attribute it to the fact that COCO's language-based metrics (e.g., CIDEr) and VQAv2's strong language priors make VLLMs insensitive to the loss of key visual details. These limitations are reflected in current community practice: our evaluation protocol mirrors that of 13 recent SOTA pruning methods, none of which report results on COCO or full VQAv2. Hence, for a comprehensive comparison with prior works, we include VQAv2's validation set in our suite.

W3 & Q5: How does pruning at deeper or multiple layers beyond ℓ = 2 impact performance and trade-off dynamics? Can additional ablations be provided to justify the pruning depth choice?

A5: Thank you for the suggestion. We clarify that a comprehensive ablation on pruning depth ($\ell=2,5,10,15,20$) is presented in Fig. 6. The results show a clear trade-off: pruning at deeper layers offers margin performance returns with notable latency increments. We chose $\ell=2$ as the default setting for two primary reasons: First, it offers a highly competitive performance-efficiency trade-off. More importantly, it aligns with the community standard for single-stage pruning methods (e.g., FastV, DART), ensuring a fair comparison in algorithmic superiority rather than conflating results with higher computational cost. Finally, multiple-layer pruning may introduce intractable high-order interactions besides the VP and PA, which are beyond the scope of our theoretical analysis. Hence, our ablations focus exclusively on this single-stage scenario.

Q6: Given the dependency of MoB on the estimation of η, what method is used to compute or approximate η in practice? Is this estimation stable and efficient?

A6: We clarify that MoB relies on an identification of the coupling pattern (i.e., 'weak' vs. 'strong') rather than a precise estimation of the $\eta$, which is much more tractable in practice. Specifically, we perform an offline analysis of a benchmark's $\eta$ distribution using Alg. 2 ($O(NLd)$, Appx. B) to identify its coupling pattern.' During inference, we simply apply the predetermined $K_{\rm p}$​ based on the coupling pattern, resulting in near-zero computational overhead at runtime.

Besides, as reported in Fig. 1(b) & 9 (Appx. D.2), the $\eta$ distributions for the two coupling patterns exhibit some overlap. This means our classification will inevitably be imperfect for samples in this overlapping region. In this case, our experimental results (Tables 1-4) show that MoB still consistently outperforms all baselines, highlighting the tolerance and stability of MoB to estimation of $\eta$.

Q7: To what extent is MoB sensitive to the choice of covering fold parameter k, especially under weak coupling conditions with long prompts? Would an adaptive mechanism be feasible here?

A7: To address your concern, we have reorganized a portion of our ablation results, focusing specifically on the sensitivity of $k$ under weak coupling conditions with long prompts:

Table 1, Ablation on fold $k$ in GQA

Table 2, Ablation on fold $k$ in TextVQA

As the tables demonstrate, MoB is not overly sensitive to the choice of $k$, particularly within a clear optimal range. For instance, in both two benchmarks, performance only varies by approximately $0.3\%$ for $k$ values between $[2, 8]$ under $\langle K=64, K_{\rm p}=32\rangle$ setting.

There is also a principled, theoretical reason for this robustness, which stems from the relationship between the covering fold $k$, the budget $K_{\rm p}$, and the length $L$ of prompt tokens $\mathcal{P}$. From covering theory, every prompt token $p\in\mathcal{P}$ is covered by at least one visual token $v\in\mathcal{V}$ under the condition $K_{\rm p}\geq kL$, thereby ensuring the performance guarantee of MoB. Therefore, as selected $k$ satisfies $k\leq K_{\rm p}/L$, the performance will remain stable.

The above analysis provides a simple yet effective heuristic: one can easily determine a robust range for $k$ based on the prompt length $L$. Based on the above analysis, we believe a more delicate mechanism for adaptively searching for a fine $k$ value for each sample may bring limited potential gains.