Each Complexity Deserves a Pruning Policy

Thank you for your detailed review and thoughtful comments. We appreciate your emphasis on our training‑free plug‑and‑play design, cognitively inspired MI, and per‑sample budgeted logistic pruning.

Q1: The introduction of neuroscience.

A1: Thank you for the insightful comment. We will revise the manuscript to simplify our prose and reduce the use of specialized neuroscience terminology. We included those details because the human brain is intrinsically skilled at reasoning and modern vision–language models demonstrate similar inference behaviors. Many approaches draw inspiration from human reasoning processes and have achieved excellent results. For example, chain‑of‑thought prompting simulates step‑by‑step human reasoning. Moreover, dual‑system, brain‑inspired approaches, known as fast and slow thinking, have been widely applied in autonomous driving and embodied intelligence domains [1–3]. Our approach aims to leverage these human visual reasoning strategies to enhance model performance. Without neuroscience‑inspired heuristics, designing an effective token pruning mechanism would be very challenging.

[1] Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoning

[2] Agentic Robot: A Brain-Inspired Framework for Vision-Language-Action Models in Embodied Agents

[3] Language‑Conditioned Robotic Manipulation with Fast and Slow Thinking

Q2: “Simple vs. Complex” in Fig. 1.

A2: Thank you for raising this point. We apologize for the confusion. Fig. 1 is intended to illustrate two observations: (1) a shared depth‑wise trend across tasks, and (2) different convergence dynamics between simple and complex questions.

1. Shared trend across tasks. As depth increases, the visual modality receives progressively less attention, as shown in Fig. 1, by Layer 31 the model’s focus almost completely shifts toward language tokens. Since this depth‑wise reduction of visual attention has been widely discussed in prior work, we did not emphasize it in the main text; we will make this point explicit to avoid misinterpretation.
2. Different convergence dynamics by task difficulty. For simple queries (Q1–Q3, object identification), attention shifts rapidly toward the target between layers 2 and 16, often by layer 4 or layer 8, and then remains stable. In contrast, for complex, reasoning-intensive queries (Q4–Q6), the model does not correctly localize the object by layer 4 and only begins to identify and stabilize the relevant region between layers 8 and 16. These divergent layer‑wise interaction patterns show that simple and complex questions follow distinct processing trajectories. This observation inspired our layer and depth aware pruning framework, which dynamically adjusts token retention to each task’s complexity and attention dynamics.
3. On saliency stability. Your observation that saliency “does not converge on the most salient region” at large depth is consistent with (1) above: late layers are increasingly language‑dominated. Our analysis focuses on the emergence and stabilization of correct visual grounding in intermediate layers, where the simple vs. complex distinction is most diagnostic.

Q3: Applicability to more recently released models.

A3: We thank the reviewer for this valuable suggestion. As shown in Tab. 2 of the manuscript, we have already validated our approach on LLaVA‑NeXT, which was released contemporaneously with Qwen2. In response, we have extended our evaluation to include Qwen2‑VL. As shown below, our method continues to deliver consistent improvements over the prior state of the art on this model, demonstrating that it generalizes robustly across the latest vision–language models.

Q4: Visualizations for numerous benchmarks.

A4: Because mutual‑information values inherently occupy a similar numerical range across datasets, the resulting curves exhibit only minor visual differences. Unfortunately, the NeurIPS rebuttal format does not permit the inclusion of new figures. We will, however, incorporate these additional plots and accompanying analysis for multiple benchmarks in the revision.

Q5: More challenging benchmarks.

A5: To address this, we have added experiments on the MMMU benchmark. As shown below, our approach continues to outperform the prior state of the art, confirming that it generalizes robustly even on more challenging, free‑form and knowledge‑heavy datasets. The full LLaVA-1.5-7B achieves an accuracy of 34.9 %. Even when retaining only 75 %, 50 % and 11.1 % of the tokens our method still surpasses the competing techniques. These findings demonstrate the strong generalizability of our approach.

Q6: Motivation of AdaPrune.

A6: We explain our motivation from the following two aspects:

• Limitations of existing approaches. Although previous pruning methods typically rely on the well‑known observation that visual attention mass decreases with layer depth, they apply the same pruning schedule to all tasks. This approach fails to adapt to each question’s unique visual–textual requirements and to the specific locations where visual evidence appears.
• Our perspective. We complement attention‑mass views with spatial dynamics of attention: how and when the model localizes relevant regions as depth increases. Inspired by human problem solving (rapid fixation for simple cases vs. search for complex ones) and supported by our visualizations, we show that simple queries stabilize early, whereas reasoning‑intensive queries exhibit delayed or shifting localization. This depth‑wise localization dynamic directly motivates a task‑adaptive pruning policy rather than a one‑size‑fits‑all schedule.

Q7: Insight from the salient map.

A7: Prior work (e.g., FastV) predominantly reports that attention magnitude on visual tokens shrinks with depth. However, in VLM pruning the focus of attention and its evolution during reasoning have received little scrutiny. Our saliency maps make this spatiotemporal relocation explicit across different question types. We then model these regularities to guide pruning decisions: stable regions trigger more aggressive pruning elsewhere, whereas shifting patterns call for conservative pruning until convergence. In the revised manuscript, we will include a clear operational definition of our saliency metric and specify the criteria that render it actionable.

I've carefully read the rebuttals and thank you for your efforts. The concerns that I was posing initially are mostly addressed. I've updated the scores towards leaning accept. I suggest that authors reflect on not only my comments but also others' ones, and please make sure the clarification on the manuscript as well.

Thank you for your detailed review and thoughtful comments. We’re grateful for your comments on the paper’s clarity and organization, the simple yet well‑motivated reframing of attention as MI, and the comprehensive experiments.

Q1: Details about which tokens are retained.

A1: We thank the reviewer for requesting more detail on token retention.

• First layer. Within the initial processing layer, the fusion of textual and visual modalities remains insufficient. As shown in Sec. D of the supplementary materials, we use Cross‑Modal Weighted Pruning. We compute each visual token’s importance score by combining its self‑attention weight with its textual attention weight. This hybrid scoring prevents discarding tokens that carry critical visual content before stronger cross‑modal interactions form.
• Subsequent layers. Visual tokens are scored by cross‑modal attention between image and text features, and we select the top $k$ tokens for downstream processing, with $k$ governed by our dynamic retention schedule. Our study targets dynamic control of the pruning budget rather than the design of ranking heuristics. To keep the analysis simple and comparable, we therefore use raw cross‑modal attention weights as the ranking criterion in all experiments. More elaborate scoring strategies reported in prior work could yield additional gains. This direction is orthogonal to our contribution and is left for future research.

Q2: Selection of $(k_0, \gamma, x_0, \beta)$.

A3: Thank you for raising this important concern. Below we provide additional detail on our hyperparameter choices and token‑selection criteria:

1. Hyperparameter mapping

   The mapping from mutual‑information score $I_q(V, T)$ to the slope $k_q$ and the inflection point $x_0^q$ is defined by

   $k_q = k_0 - \gamma\,I_q(V, T),\quad x_0^q = x_0 + \beta\,I_q(V, T).$

   To demonstrate robustness, we fix these values across all experiments (See below for parameter selection). Specifically we set

   $k_0 = 0.4,\quad \gamma = 0.3,\quad x_0 = 15,\quad \beta = 0.3.$

   We apply only one clamp, enforcing $k_q \ge 0$ so that the selection process can only decrease the token number and never increase it. We do not apply any calibration to the mutual‑information values.

2. Token‑selection criteria

   Visual tokens are scored by cross‑modal attention between image and text features, and we select the top $k$ tokens for downstream processing, with $k$ governed by our dynamic retention schedule. Our study targets dynamic control of the pruning budget rather than the design of ranking heuristics. To keep the analysis simple and comparable, we therefore use raw cross‑modal attention weights as the ranking criterion in all experiments. Although more sophisticated scoring mechanisms described in prior work may yield further gains, such enhancements fall outside the scope of the present study and are reserved for future investigation.

3. Robustness across hyperparameters

   • Parameter selection strategy. 1) Curve-based criteria. As shown in Fig. 2, the retention curve must clearly distinguish easy from hard samples. We therefore choose parameters that provide sufficient dispersion, avoiding curves that concentrate around the midpoint or flatten at the extremes, so that the pruning schedule remains balanced. 2) Empirical validation**.** A small series of preliminary experiments yielded the following hyperparameters without extensive tuning:

     $k_0 = 0.4,\quad \gamma = 0.3,\quad x_0 = 15,\quad \beta = 0.3.$

     Specifying each value to one decimal place demonstrates that no overfitting occurred on any particular dataset.

   • Supplementary experiments. To validate the robustness of these settings, we ran additional experiments under alternative hyperparameter configurations. Although some variants produced slightly improved results (see the table below), we elected to retain the original fixed values in order to ensure fairness and consistency across all datasets.

     Configuration	TextVQA(token=64)↑	TextVQA(token=128)↑	GQA(token=64)↑	GQA(token=128)↑
     $k_0=0.3, \gamma=0.2, x_0=16, \beta=0.5$	57.2	57.6	57.3	59.4
     $k_0=0.5, \gamma=0.4, x_0=14, \beta=0.2$	56.9	57.3	57.6	59.6
     $k_0=0.4, \gamma=0.3, x_0=15, \beta=0.3$(default)	57.1	57.4	57.7	59.9

Q3: Which layer is this Mutual Information computed?

A3: We appreciate the reviewer's valuable suggestion.

1. Optimal Layer Selection for MI Estimation. Thank you for raising this important and insightful concern. Estimating mutual information too early can indeed introduce inaccuracies, which is why we perform MI estimation beginning at the second layer of the network. To address your question, we have conducted additional experiments measuring MI at various layers. The results, summarized in the table below, reveal two key trends. First, estimating MI too early (at layer 1) injects spurious signals into the pruning schedule, causing the model’s accuracy to suffer. Second, delaying MI estimation until very late layers front‑loads an excessive share of the fixed compute budget, which also diminishes overall performance. Estimating MI at layer 2 therefore achieves the optimal balance between reliable signal extraction and efficient budget utilization. Notably, in layers where MI is not estimated, we employ a linear pruning schedule in our experiments.

   Layer	TextVQA(token=64)↑	TextVQA(token=128)↑	GQA(token=64)↑	GQA(token=128)↑
   1	56.4	56.8	56.5	59.1
   2	57.1	57.4	57.7	59.9
   3	56.9	57.2	57.4	59.5
   4	56.6	57.0	57.2	59.4

2. Cross‑Modal Hybrid Token Scoring Mechanism. Within the first layer, the fusion of textual and visual modalities remains insufficient. As shown in Sec. D of the supplementary materials, we use Cross‑Modal Weighted Pruning. We compute each visual token’s importance score by combining its self‑attention weight with its textual attention weight. This hybrid scoring prevents discarding tokens that carry critical visual content before stronger cross‑modal interactions form.

Q4: Limitations: Important unaddressed considerations.

A4: We agree that our “Conclusions and Limitations” section can be strengthened by more explicitly acknowledging key shortcomings. We have identified and will add the following limitations to the revised manuscript:

1. Lack of a token recall mechanism. Although token importance generally decreases with decoder depth, our analysis in Fig. 1 reveals that deeper layers sometimes retain tokens that are more critical than those in shallower layers. This finding motivates the development of a mechanism to recover valuable tokens that were discarded in early layers.
2. Application to video-based VLMs. Our pruning method is, in principle, directly transferable to video vision-language models, but we have not yet introduced any temporal modeling. Incorporating temporal information into our token pruning framework represents a meaningful direction for future research.

We will incorporate these points into the Limitations section in the revision.

Thank you for your detailed review and thoughtful comments. Thank you for noting the strong motivation and the novelty of using visual–textual mutual information to adapt pruning to task complexity rather than relying on fixed, hand‑crafted policies.

Q1: Why the Budget-Constrained Logistic Retention is designed as Eq.5?

A1: We adopt a logistic retention function for three key reasons:

1. Neuro‑inspiration: As discussed in lines 208–210 of the main text, human visual reasoning over time follows an S‑shaped information accumulation curve, which closely matches the profile of a logistic function.
2. Proven efficacy: The logistic curve, often referred to as the logistic sigmoid, is extensively employed in contemporary LLM and VLM research. It underlies sigmoid-based language–image pre-training losses [1], serves as the gating function in mixture-of-experts routing [2], and plays a central role in various optimization algorithms [3].
3. Empirical validation: In Tab.4 (b), we compare linear, tanh, exponential, and logistic mappings. The logistic formulation consistently achieves the best balance between adhering to the token budget and maintaining task performance, justifying its selection.

[1] Sigmoid Loss for Language-Image Pre-Training, ICCV 2023

[2] Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts, NeurIPS 2024

[3] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS 2023

Q2: The reason for retention rate for easy and hard samples.

A2: We appreciate the reviewer’s insight. The observed retention patterns arise for three reasons:

1. Fixed overall budget. To satisfy our compute‑cost constraint, we enforce the same total token budget for both easy and hard samples. The curves therefore show the optimal pruning schedule under an identical compute budget.
2. Behavior on easy samples. Easy samples allow the model to rapidly locate in the relevant regions, so we can apply aggressive pruning in the earliest layers. After pruning, the model repeatedly attends to the retained tokens to guarantee correctness. (Note that if the compute budget were unconstrained, one could introduce further deep pruning to reduce computation even more for these easy cases.)
3. Behavior on hard samples. Difficult images require an initial exploratory phase in which the model must survey many tokens to locate the target. Hence, we delay pruning in early layers (high retention).

Q3: Improvements of experiments.

A3: We thank the reviewer for these valuable suggestions:

1. More complex Image caption task for evaluation.

   We extend our pruning method to the COCO Caption dataset using the LLaVA 1.5-7B model and evaluate performance with the CIDEr and SPICE metrics. The unpruned baseline achieves a CIDEr score of 1.048. When retaining only 75%, 50% and 11.1% of the tokens, our approach continues to outperform existing methods on both metrics. Notably under heavy pruning at 11.1%, the gains are especially pronounced compared to others. These results underscore the strong generalization ability of our method.

   Method	75%	50%	11.1%
   FasterVLM（ICCV 25）	1.032	0.997	0.872
   PyDrop (CVPR 25)	0.999	0.981	0.739
   Ours	1.039	1.028	0.946

2. Inclusion of VTW, Turbo, and Folder.

   We appreciate the reviewer’s recommendation to examine VTW, Turbo and FOLDER. In Tab. 2 of the supplementary material, we include a comparison with VTW. We have conducted new experiments comparing AdaPrune with Turbo and FOLDER, and we will include a discussion of all three methods in our related‑work section. As shown in table below, AdaPrune achieves superior accuracy, again demonstrating its robustness and efficiency. All data in the table are taken directly from the original FOLDER publication.

   Method	Retaining Ratio	MMBench-en↑	MMMU↑	MME↑
   LLaVA1.5-7B	100%	62.8	32.2	1339
   Turbo	34%	60.1	24.7	1302
   FOLDER	34%	61.4	31.3	1350
   Ours	34%	64.3	34.2	1433

3. Absolute performance of compared baseline model.

   We have updated Tab. 3 to include the absolute scores of the baseline model (100% token retention). For example, the baseline accuracy is 41.49%, and AdaPrune achieves 46.15% at only 25% token usage. Including these numbers helps clarify that our relative gains correspond to a small absolute drop in accuracy while significantly reducing computation.

   Method	26/128 (20%)	32/128 (25%)	38/128 (30%)	45/128 (35%)	51/128 (40%)
   FasterVLM	22.84	21.91	19.58	18.88	20.51
   SparseVLM	34.97	38.93	39.63	40.56	41.96
   PyramidDrop	39.39	41.03	39.86	40.78	41.72
   Ours	40.09	46.15	44.29	43.59	43.36

Thank you for your thoughtful comments and for the time you have invested in our work. We have posted our response and have successfully resolved the issues raised by the other two reviewers, who have in turn increased their scores. We hope that the above clarifications and the additional experiments sufficiently addressed your concerns. If any questions remain unresolved, please let us know. We would be grateful to clarify further and are happy to discuss specific points.

Thank you for your detailed review and thoughtful comments. We appreciate your recognition of our clear writing, the simple training‑free MI signal that adapts pruning per input, the exact‑budget logistic retention curve, and the broad evaluations on LLaVA‑1.5/NeXT and a driving VLM.

Q1: More detail on hyperparameter choices and token-selection.

A1: Thank you for raising this important concern. Below we provide additional detail on our hyperparameter choices and token‑selection criteria:

1. Hyperparameter mapping

   The mapping from mutual‑information score $I_q(V, T)$ to the slope $k_q$ and the inflection point $x_0^q$ is defined by

   $k_q = k_0 - \gamma\,I_q(V, T),\quad x_0^q = x_0 + \beta\,I_q(V, T).$

   To demonstrate robustness, we fix these values across all experiments (See below for parameter selection). Specifically we set

   $k_0 = 0.4,\quad \gamma = 0.3,\quad x_0 = 15,\quad \beta = 0.3.$

   We apply only one clamp, enforcing $k_q \ge 0$ so that the selection process can only decrease the token number and never increase it. We do not apply any calibration to the mutual‑information values.

2. Token‑selection criteria

   Visual tokens are scored by cross‑modal attention between image and text features, and we select the top $k$ tokens for downstream processing, with $k$ governed by our dynamic retention schedule. Our study targets dynamic control of the pruning budget rather than the design of ranking heuristics. To keep the analysis simple and comparable, we therefore use raw cross‑modal attention weights as the ranking criterion in all experiments. Although more sophisticated scoring mechanisms described in prior work may yield further gains, such enhancements fall outside the scope of the our study and are reserved for future investigation.

3. Robustness across hyperparameters

   • Parameter selection strategy. 1) Curve-based criteria. As shown in Fig. 2, the retention curve must clearly distinguish easy from hard samples. We therefore choose parameters that provide sufficient dispersion, avoiding curves that concentrate around the midpoint or flatten at the extremes, so that the pruning schedule remains balanced. 2) Empirical validation A small series of preliminary experiments yielded the following hyperparameters without extensive tuning:

     $k_0 = 0.4,\quad \gamma = 0.3,\quad x_0 = 15,\quad \beta = 0.3.$

     Specifying each value to one decimal place demonstrates that no overfitting occurred on any particular dataset.

   • More experiments. To validate the robustness of these settings, we ran additional experiments under alternative hyperparameter configurations. Although some variants produced slightly improved results (see the table below), we elected to retain the original fixed values in order to ensure fairness and consistency across all datasets.

     Configuration	TextVQA(token=64)↑	TextVQA(token=128)↑	GQA(token=64)↑	GQA(token=128)↑
     $k_0=0.3, \gamma=0.2, x_0=16, \beta=0.5$	57.2	57.6	57.3	59.4
     $k_0=0.5, \gamma=0.4, x_0=14, \beta=0.2$	56.9	57.3	57.6	59.6
     $k_0=0.4, \gamma=0.3, x_0=15, \beta=0.3$(default)	57.1	57.4	57.7	59.9

Q2: Early-layer MI estimates risks misjudging.

A2: Thank you for raising this important and insightful concern.

1. Optimal Layer Selection for MI Estimation. Estimating mutual information too early can indeed introduce inaccuracies, which is why we perform MI estimation beginning at the second layer of the network. To address your question, we have conducted additional experiments measuring MI at various layers. The results, summarized in the table below, reveal two key trends. First, estimating MI too early (at layer 1) injects spurious signals into the pruning schedule, causing the model’s accuracy to suffer. Second, delaying MI estimation until very late layers front‑loads an excessive share of the fixed compute budget, which also diminishes overall performance. Estimating MI at layer 2 therefore achieves the optimal balance between reliable signal extraction and efficient budget utilization. Notably, in layers where MI is not estimated, we employ a linear pruning schedule in our experiments.

   Layer	TextVQA(token=64)↑	TextVQA(token=128)↑	GQA(token=64)↑	GQA(token=128)↑
   1	56.4	56.8	56.5	59.1
   2	57.1	57.4	57.7	59.9
   3	56.9	57.2	57.4	59.5
   4	56.6	57.0	57.2	59.4

2. Cross‑Modal Hybrid Token Scoring Mechanism. Within the first layer, the fusion of textual and visual modalities remains insufficient. As shown in Sec. D of the supplementary materials, we use Cross‑Modal Weighted Pruning. We compute each visual token’s importance score by combining its self‑attention weight with its textual attention weight. This hybrid scoring prevents discarding tokens that carry critical visual content before stronger cross‑modal interactions form.

Q3: How AdaPrune scales to much larger models, video inputs, or streaming scenarios and Tab. 4 in supp.

A3: We appreciate the reviewer’s suggestion for scalability and future extensions to video and streaming.

1. Scales to much larger models.

   AdaPrune is entirely model‑agnostic and requires no architectural modifications to operate on larger vision‑language models. In addition to the LLaVA‑13B results presented in Supplementary Table 4, we also evaluated AdaPrune on CogVLM-17B [1] by pruning 90 % of the tokens during inference. As shown in table below, AdaPrune consistently delivers significant inference‑cost reductions with negligible performance loss across both 7B, 13B and 17B variants, demonstrating robustness to model scale.

   Method	Retaining Ratio	VQAv2↑	VizWiz↑	SQA↑
   CogVLM-17B	100%	80.9	49.6	68.4
   FastV (ECCV 24)	10%	74.2	42.9	63.5
   FasterVLM（ICCV 25）	10%	74.6	46.6	68.2
   Ours	10%	75.7	48.3	68.4

​ [1] Cogvlm: Visual expert for pretrained language models, NeurIPS 2024

2. Video and streaming inputs. Thank you for this suggestion. Although our method is specifically designed for image–text VLMs, we believe its extension to video and real‑time streaming is highly promising and could bring significant benefits. The key challenge in this setting is maintaining temporal consistency in token selection, because pruning each frame independently may lead to flickering or unstable focus regions. We anticipate that simple extensions, such as (a) smoothing attention weights over successive frames to carry forward high‑importance tokens and (b) dynamically allocating a per‑frame token budget based on motion or scene change estimates, will address these issues with minimal changes to our framework. We plan to explore these temporal adaptations in future work.

Q4: Assessing AdaPrune’s Robustness to MI Estimation Noise.

A4: Thanks to the reviewer for raising the need to evaluate how MI estimation noise affects AdaPrune.

We acknowledge that errors in the mutual information estimate can influence pruning decisions. When MI is overestimated, key tokens may be pruned too early. When MI is underestimated, the token set may be larger than necessary. Despite these effects, the impact on overall accuracy remains limited. Tab. 4(a) shows that a fixed logistic retention schedule ignoring per‑layer MI still achieves competitive performance.

To measure sensitivity, we injected random perturbations of ±10 %, ±20 % and ±50 % into the MI values before computing the retention schedule and reran our experiments. The results show that perturbations in the 10–20% range cause only minor degradation in accuracy and only the large 50% noise produces a more noticeable drop. These findings confirm that AdaPrune remains robust, only degrading when MI estimation noise is substantial.

Q5: More implementation details of compute MI.

A5: We compute mutual information exactly as in Section 3.3 of the paper. Specifically, we first extract the attention map between the visual and textual modalities and normalize its entries to obtain the joint distribution $p(v_i, t_j)$. Treating the marginal distributions as $p(v_i)=\sum_j p(v_i,t_j)$ and $p(t_j)=\sum_i p(v_i,t_j)$, we then compute

$I(V, T) = \sum_{i=1}^{N_v}\sum_{j=1}^{N_t} p(v_i, t_j)\,\log\frac{p(v_i, t_j)}{p(v_i)\,p(t_j)}.$

This directly measures the amount of information shared between visual and textual tokens based on their attention weights.