Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention

We sincerely thank Reviewer cqzx for the insightful comments.

---

Q1: Can you provide detailed information on how spatial crops, crop numbers, and related hyperparameters are selected? Additionally, could you include ablation studies that show how these choices impact the reproducibility and performance of your approach?

We show performance when we vary the total number of crops from 4 to 24, and we observe that varying the number of crops in the range of 4-16 does not have a significant effect on performance, but if the number of patches is too large (>20), it causes a decrease in performance.

where the numbers indicate the percentage of performance relative to the raw performance.

For hyperparameters, we usually set allocation sharpness $\tau=1$, and set the same value in all experiments.

---

Q2: What is the rationale behind your specific top-k token selection score? How does this formulation compare empirically or theoretically to alternative selection strategies or scoring methods?

Our objective is $\mathcal{H}^c = \gamma_c \mathcal{V}^c + \mathcal{A}^c$. Following your suggestion, we divided the objective into local saliency $\mathcal{V}$ and global relevance $\mathcal{A}$, and we compare against pure local saliency, global relevance, and our HoloV. The results when retaining 64 tokens are as follows,

The results demonstrate that HoloV's strategy is reasonable. About whether more sophisticated selection strategies could lead to further improvements, we are researching a text-based approach to further improve the performance of HoloV.

---

Q3: Have you evaluated the sensitivity and robustness of your adaptive holistic token allocation procedure, including the effects of discretization and the sharpness parameter? Could you discuss or provide results comparing your allocation approach to other possible methods?

While it needs setting of the sharpness parameter, we set $\tau=1$ in all experiments, and we also tried setting other values, and the results show that the model is not sensitive to this hyperparameter. For the effectiveness and robustness of this adaptive allocation, we vary the total number of crops from 4 to 16, and we observe that varying the number of crops does not have a significant effect on performance.

---

Q4: Would it be possible to use HoloV as a replacement for the Q-Former module in models such as InstructBLIP, or to integrate HoloV within the Q-Former family of architectures? If so, could you discuss any practical considerations, challenges, or potential benefits of applying HoloV in this context? It would be helpful to understand whether HoloV is compatible with Q-Former-based models and how its holistic token pruning strategy might interact with or improve these architectures.

To integrate HoloV within the Q-Former family of architectures, we conducted experiments on Qwen2.5-VL.

These results demonstrate that HoloV achieves superior performance.

---

Q5: Could you report practical efficiency metrics such as multiply-accumulate operations (MACs), wall-clock latency, and memory usage on different hardware platforms, including less powerful or older devices and different devices (GPUs with different floating point operations)? It would be helpful to see evidence of real-world efficiency beyond theoretical FLOPs reductions.

We reported practical efficiency metrics on 4060Ti GPU (16GB), the results are as follows,

---

Q6: Can you present empirical results or ablation studies that isolate and quantify the actual contribution of visual context refetching (VCR) in your framework? In particular, how is the injection ratio chosen, and what impact does VCR have on uncertainty reduction or downstream performance?

Sure. We added experiments about the isolated contribution of visual context refetching (VCR) in our framework and under different injection ratios.

For the injection ratio, we often set it to 0.15, and this strategy can effectively contribute to the performance improvements.

---

Q7: Do you have results or ablation studies demonstrating how HoloV performs with a wider variety of visual encoders, such as SigLIP, DINOv2, CLIP-ResNet,S2, Mamba, ConvNeXt, etc? Would you consider evaluating whether your method generalizes to models beyond the LLaVA family?

We have not conducted ablation studies with a wider variety of visual encoders as time is too short and this needs pretraining these visual encoders (most MLLMs' visual encoders are ViT framework), in the future we will take time to research such influence. Following your suggestion, we have additionally included experiments with Qwen2.5-VL (beyond the LLaVA family), and the results on multiple benchmarks demonstrate HoloV's effectiveness. (Table in response to Q4)

---

Q8: Paper Formatting Concerns: Several tables and figures in the paper do not follow the official NeurIPS template and formatting guidelines. In particular, Figure 5, Figure 6, Figure 9, Table 3, and Table 4 stand out as not being consistent with the expected style. I recommend revising these figures and tables to conform with the NeurIPS template.

We have reformatted all figures and tables to match the NeurIPS style guidelines—single-column width for tables, consistent font sizes, correct caption placement, and vector‐format graphics.

Q5: In computing, the execution time of a piece of code depends on many factors: hardware factors like the CPU and GPU, and software factors such as OS versions and Python dependencies, among others. Therefore, quantitatively, having execution time alone is not a universal metric to consider. That’s why I asked about FLOPS and MACs using Holo and the baseline without Holo. You don’t need to run multiple comparisons with various token lengths; you can use a smaller number of tokens, which is the fastest/efficiency can be backed up with numbers.

For latency, you should run one example only and use profiling like this:

start = torch.cuda.Event(enable_timing=True)
end   = torch.cuda.Event(enable_timing=True)

start.record()
output = model(input)
end.record()

# Wait for all kernels to finish
torch.cuda.synchronize()
print(start.elapsed_time(end), "ms")

For MACS and FLOPS:

We sincerely thank Reviewer h9EW for the constructive comments on our work. Below, we will address each of your questions and concerns.

---

Q1: The fixed spatial crop partitioning method limits the system's applicability in complex scenes where semantic regions may be non-rectangular or cross boundaries. This can result in fragmented semantic units, reducing the model’s generalizability.

Actually, the quota of a crop is dynamically allocated, thus, this would not affect flexible allocation based on the density or complexity of the content in the image, which could reduce fragmentation and improve model performance in more intricate scenes.

---

Q2: There is also a sensitivity to hyperparameters like the sharpness factor (τ) and scaling factor (γ), which require manual tuning. The lack of a unified optimization strategy for varying resolutions or scene complexities increases deployment costs and limits real-world applicability.

We acknowledge that τ (sharpness factor) requires manual setting, but in practice, these hyperparameters do not have a significant effect on the results. For example, the sharpness factor (τ) is always set to 1. For $\gamma_c$ (scaling factor), it is determined by $\mathbb{E}[\|\mathcal{A}^c\|]/\mathbb{E}[\|\mathcal{V}^c\|]$.

---

Q3: While HoloV mitigates Positional Bias, it still struggles in extreme conditions such as severe lighting, occlusions, or fast-moving objects. These challenges lead to potential information loss, as seen in MSRVTT-QA experiments where pruning errors affected dynamic object understanding.

Thank you for your recognition of our work. We agree that extreme conditions like severe lighting, occlusions, and fast-moving objects pose significant challenges for HoloV and can lead to information loss. In response, we are considering temporal pruning, i.e., tracking dynamic objects across frames, pruning static duplicates, to better capture dynamic changes and reduce pruning errors in such conditions. This strategy would improve the robustness of HoloV in challenging scenarios and dynamic object understanding, where traditional image models may struggle to capture the full range of information.

---

Q4: How does the granularity of crop partitioning (e.g., number of crops) affect pruning efficiency? For high-resolution images (e.g., LLaVA-NeXT's 2,880 tokens), should the number of crops be dynamically adjusted?

In fact, the granularity of pruning partitions does not affect pruning efficiency, as the retained visual markers are determined by pruning quotas, and the quota in a crop is calculated by the informativeness of intra-crop visual tokens, which is dynamic, the total pruning quotas would not be changed. For high-resolution images, we agree that adjusting the number of crops dynamically could be beneficial. For instance, using a lower number of crops for highly detailed areas (such as objects or regions of interest) and a higher number of crops for less detailed regions (such as background areas) could improve pruning performance.

Further, we show performance when we vary the total number of crops from 4 to 16, and we observe that varying the number of crops does not have a significant effect on performance.

where the numbers indicate the percentage of performance relative to the raw performance.

---

Q5: When an LLM has severe language bias, could HoloV's visual pruning indirectly exacerbate hallucinations? For example, if the language model misjudges an object's existence, might HoloV retain irrelevant visual tokens that reinforce this error?

We recognize that when an LLM has severe language bias (e.g., in tasks involving object recognition or reasoning), it may hallucinate or misjudge the existence of certain objects. If this occurs, visual pruning could indeed exacerbate the issue by retaining irrelevant visual tokens. However, in the experiments, especially in POPE, which is a hallucination-related benchmark, compared with SOTA methods, our HoloV achieves the best performance, as shown in the table.

The results demonstrate that HoloV's visual pruning has minimal impact on exacerbating hallucinations.

In the future, we aim to explore cross-modal consistency checks, where visual tokens are only retained if they align with the language model's predictions. If the LLM's prediction is highly uncertain, we would retain more visual tokens associated with that object, potentially reducing hallucinations.

We appreciate the reviewer’s thoughtful feedback on these critical aspects of HoloV. We look forward to continuing to refine HoloV for real-world applicability.

We sincerely thank Reviewer 4HaK for the constructive comments on our work. We are very grateful to the reviewer for recognising the novelty of our idea and the richness and rationality of our experiments.

---

Q1: The evenly partitioned crops may not be flexible enough and could introduce potential spatial bias.

Thanks. Although crops are partitioned evenly, the quota of a crop is dynamically allocated, thus, this would not affect flexible allocation based on the density or complexity of the content in the image, which could reduce fragmentation and improve model performance in more intricate scenes.

To further show the advantages of our method, we further present our results on MM-Vet benchmark as follows,

---

Q2: I would like to know whether HoloV can still achieve consistent strong performance on models using native resolution (e.g., Qwen2.5-VL). Adding such results would allow HoloV to cover all mainstream VLM visual encoding forms, further enhancing the persuasiveness of the results.

We agree with the importance of having HoloV cover all major VLMs, and based on your suggestion we have additionally included experiments with Qwen2.5-VL, and the results on multiple benchmarks demonstrate HoloV's effectiveness.

These results demonstrate that HoloV achieves superior performance.

We sincerely thank Reviewer MYQs for the constructive comments on our work. We promise to revise the paper.

---

Q1: some concepts in the paper may need revision. 

• Line 51 & Line 65: “Semantic connectivity”  The concept of “semantic connectivity” means the cross-region interactions in semantic space, e.g., A on/under/in B, such object relations, retaining only "highlighted tokens" would lose this connectivity. We have clarified in the revision paper.  To further substantiate this concept, we have added to Figure 3 heatmaps in the revision paper for two representative images (pedestrian and vehicle), showing pairwise attention strengths between adjacent and distant visual tokens. 
• Line 150 & Line 157-159: Inconsistency in “attention dispersion” presentation.  Sorry for miswriting, "attention dispersion" is a counter concept that more visual tokens take the majority of the attention. Under this concept, compared to [CLS] attention (the top 20% of visual tokens account for 80% of the total attention), text-vision attention tends to be dispersed over more visual tokens, e.g., the top 20% of visual tokens account for only 40% of the total attention, consistent with "attention dispersion". We have revised this mistake in the revision. 
• Line 164 - 165: the concept of "global contextual cohesion" is not necessary.neng  Thanks for your nice suggestion. We agree, our initial idea means not over-reliance on attention scores, as it disrupts spatial semantic relationships, considering global semantics could mitigate this issue. We have revised this concept in the revision.

---

Q2: Line 134 - 135 "As shown in Fig. 4 left, adjacent tokens with similar visual features frequently receive comparable attention scores" It seems difficult to reach such conclusion from Fig. 4 left. Please elaborate more on it.

Take FastV's distribution map of visual token attention as an example, many tokens in regions characterized by flat backgrounds, such as sky, grass, and trees with similar visual features, receive comparable attention scores. When using the top-k strategy to select retained visual tokens, it leads to semantic redundancy, i.e., retaining many visual tokens with semantics of the same objects.  Further, we added more cases and a statistical analysis to elaborate on this redundancy in the revised Appendix.

---

Q3: Line 131 "we focus on the attention received by visual tokens from the visual [CLS] token" Seems there is a limitation of the only focus on attention from the CLS token. I wonder if the mutual attention between visual token needs to be considered.  We clarified that our primary goal was to study how [CLS] aggregates global visual information, while mutual attention between visual tokens was included in self-attention at ViT, and we don't need this mutual attention between visual tokens to select key visual tokens.

---

Q4: Sec 4.1 Issues in the formula (1) need to show dimension for the matrices and vectors (2) eq.2 should have $\sum_j$ instead of $\sum$ (3) $\mathcal{H}^c$ is never used (4) $\Omega_c$ is not well explained (5) In eq. 5, why minimizing global diversity $\mathcal{V}^c$? Do we want similar patterns in each crop?

R(1), we actually have listed dimension of $\mathbf{Z}_v^c \in \mathbb{R}^{M\times d}$, further we follow your comment to show clear dimension, in the formula (1) $\mathbf{S}^c = (\mathbf{1}-\mathbf{I}_M)\odot\mathbf{Z}_v^c {\mathbf{Z}_v^c}^\top$, where $\mathbf{Z}_v^c {\mathbf{Z}_v^c}^\top \in \mathbb{R}^{M\times M}$, and $(\mathbf{1}-\mathbf{I}_M) \in \mathbb{R}^{M\times M}$, thus $\mathbf{S}^c \in \mathbb{R}^{M\times M}$.

R(2), thanks for your kind suggestion, we have revised from $\sum$ to $\sum_j$.

R(3), $\mathcal{H}^c=\gamma_c \mathcal{V}^c + \mathcal{A}^c$ indicates our holistic attention that is calculated through a balanced scoring mechanism combining contextual diversity and attention saliency, in the code we actually use it to calculate the crop importance weights in formula (4), as there is a writting mistake, we have revised it to $w_c = \frac{(\frac{1}{M}\sum_{t=1}^M \mathcal{H}_t^c)^\tau}{\sum^{\mathcal{C}} (\frac{1}{M} \hat{\mathcal{H}}^{c'})^\tau}$,

where $\hat{\mathcal{H}}^{c'}=\sum_{t=1}^M \mathcal{H}_{t}^{c'}$.

R(4), $\Omega_c$ means the objective has a quota constrain, i.e., $|\Omega_c| = q_c$

R(5), $\mathcal{V}_i^c$ indicates that $i$-th token has diverse connections with others, our objective is to retain those with rich semantics tokens within a crop. To avoid ambiguity, we revised formula (5) to more clearly express the idea of the paper and conform to the actual implementation of the code.

---

Q5: Typo: Table 3 Avgerge -> Average

We have revised our paper accordingly, the typo "Avgerge" is revised to "Average" in Table 3.

We sincerely appreciate your valuable suggestions again.

Dear Respected Reviewer MYQs,

I hope you are well. With the discussion period ending in less than one day, we wish to confirm all your concerns have been addressed satisfactorily.

Please feel free to share any additional feedback—your insights are invaluable, and we are eager to refine our work accordingly.

Thank you sincerely for your time and effort in reviewing our paper.