Glance2Gaze: Efficient Vision-Language Models from Glance Fusion to Gaze Compression

Thank you for your valuable feedback.

Here we address the points you mentioned in the weaknesses part.

1. We appreciate the reviewer’s emphasis on reproducibility and the interest in accessing our code. Our code will be released upon paper’s acceptance.

2. We clarify that among the four video-QA datasets in Table 3 of the main text, only ActivityNet shows a slight performance gain for Glance2Gaze over naive Video-LLaVA. On TGIF and the other datasets, Glance2Gaze generally underperforms, aligning with the expectation that compression reduces available information and may impair performance. However, video data often contain significant redundancy and noise within visual tokens, which can hinder model comprehension. The compression applied by Glance2Gaze aims to distill salient visual information by removing redundant or noisy tokens. This simplification can yield a clearer, more focused representation, enabling better feature capture and, occasionally, improved performance on specific datasets or samples. Therefore, the occasional performance improvement post-compression is plausible and consistent with the data characteristics and model behavior.

3. Thanks for your insightful comment. We conducted additional experiments incorporating a larger model following Dense Connector for fair comparison. Specifically, we employed SigLIP-so-384px as the visual encoder alongside the Yi-34B large language model to evaluate the effectiveness of our Glance Fusion method at scale. Due to resource limitations, we utilized the LoRA fine-tuning technique for training the projector and the LLM, with lora_r and lora_alpha set to 128 and 256, respectively, consistent with Dense Connector’s settings. Given that SigLIP comprises 27 ViT layers, we applied Glance Fusion at layers [7, 14, 21, 27]. The results, summarized in the table below, indicate that Glance Fusion significantly enhances performance in larger models, surpassing Dense Connector.

   Method	training	GQA	VQA v2	SQA	TextVQA	POPE
   baseline (SigLip-Yi-34B)	LoRA	62.4	81.1	78.8	64.7	81.4
   + Dense Connector	LoRA	63.9 (+1.5)	-	80.5 (+1.7)	66.7 (+2.0)	80.5 (-0.9)
   + Glance Fusion	LoRA	64.3 (+1.9)	83.5 (+2.4)	81.9 (+3.1)	67.5 (+2.8)	82.6 (+1.2)

Thank you for your thoughtful comments and suggestions.

Here we address the points you mentioned in the weaknesses part.

1. We appreciate reviewer's attention on training cost, FLOPs computation and inference performance.

   1). Here we provide comprehensive tables presenting training cost, inference latency, prefilling time, and throughput across various frameworks and compression settings to offer a clear picture of computational costs.

   Method	Training GPU Hours	Inference TFLOPs	Latency (ms)	Cuda Time (ms)	Throughput
   LLaVA-1.5-7B	104	10.07	185.9	115	28.754
   Glance2Gaze (33% FLOPs)	72	3.36	133.2	66	40.022
   Glance2Gaze (22% FLOPs)	60	2.22	109.2	32	51.351
   Glance2Gaze (11% FLOPs)	43	1.11	100.2	22	56.82

   Method	Training GPU Hours	Inference TFLOPs	Latency (ms)	Cuda Time (ms)	Throughput
   LLaVA-Next-7B	366	53.83	475.2	313	10.884
   Glance2Gaze (22% FLOPs)	242	11.84	262.0	100	19.741
   Glance2Gaze (11% FLOPs)	173	5.53	191.9	90	30.719
   Glance2Gaze (5% FLOPs)	87	2.67	151.2	49	34.121

   Method	Training GPU Hours	Inference TFLOPs	Latency (ms)	Cuda Time (ms)	Throughput
   Video-LLaVA	297	37.38	739.6	342.2	21.599
   Glance2Gaze (6% FLOPs)	151	2.24	484.3	125.1	35.095

   2). Following common practice in visual pruning literature (e.g., VisionZip, Pdrop, SparseVLM), we focus FLOPs measurement primarily on the computation related to visual tokens within the LLM. This is because pruning techniques selectively reduce computation in the LLM's visual token processing, while the visual encoder’s computational cost remains fixed and thus does not affect the pruning evaluation. Here we also present a detailed breakdown of FLOPs across different components of the vision-language model, including the visual encoder.

   Glance2Gaze (11% FLOPs)	in Visual Encoder	in Glance Fusion	in LLM
   FLOPs	0.38 T	0.03 T	1.1 T
   proportion	0.2517	0.0199	0.7285

2. Thank you for the helpful suggestion. We will revise the relevant equations and FLOPs computations in the main text and supplementary to improve clarity and transparency.

Then here are the answers to the Questions part.

1. Thank you for pointing this out. We acknowledge that the reuse of the same symbol in Equation (9) as well as the reuse from Equation (8) could lead to confusion regarding the semantics of the visual tokens at different stages of the pipeline. We will carefully revise the notation in the final paper.

2. Please see the answers to Weakness one for more comprehensive tables presenting training cost, inference latency, prefilling time, and throughput. Regarding the discrepancy between FLOPs reduction and wall-clock speedup, this stems from the well-established distinction between theoretical computational savings and practical runtime acceleration. FLOPs reductions do not translate linearly to real-world speedups due to factors such as memory access patterns and hardware constraints. Nonetheless, achieving a 6.39× speedup represents a substantial improvement in inference latency, underscoring the practical effectiveness of our approach.

3. We thank the reviewer for highlighting this point and apologize for any confusion caused in FLOPs computation of visual tokens.

   1). Equation (2) aims to compute the total computational cost of processing visual tokens within the decoder, which consists of two parts:

   • the standard forward pass through the decoder (first term),
   • the additional cost introduced by the Gaze Compression module (second term).

   We will revise the title to “FLOPs within LLM (Visual Tokens)”, to reflect that both standard forward and gaze compression contribute to the total cost.

   2). Regarding the variable $n$ in the first term of Equation (2), your interpretation is correct. As clarified in Line 206 of the main text, $n$ denotes a variable that remains constant prior to layer $r$ and takes the value of $p_l$ thereafter.

   3). We appreciate the reviewer pointing out the omission. The FLOPs computation for Glance Fusion should indeed include the cost of projection layers for both text and visual inputs. In Glance Fusion, the computational cost should include $S$ projection layers for text mapping and $S$ projection layers for visual mapping, which bring $SNd_vd_t$ and $SMd_t^2$ FLOPs respectively. After correction, the FLOPs for the Glance Fusion stage be 0.03T. Thus, the ratio of computation between Glance Fusion and the LLM is $0.03/1.1=0.0272$. We will update Equation (1) and the related text accordingly.

4. We compute visual token importance by averaging instruction-to-image attention scores extracted from the decoder's attention matrices. Specifically, we isolate the instruction-to-visual attention submatrix and calculate the mean attention each visual token receives from all instruction tokens, yielding a scalar importance score per token. Tokens are ranked accordingly, and those with the lowest scores are pruned based on a predefined ratio. Pruned tokens and their residuals are completely excluded from subsequent layers and computations, thus exerting no influence on downstream activations.

5. Your understanding is correct. The "Layer-specific Query" bars in Figure 4 do include the Glance Fusion stage. The only difference from the full Glance2Gaze model is that the query embeddings are not shared across layers.

6. Regarding the small comments, we thank the reviewer for pointing out grammatical and formatting errors, and we will address these in the final paper. Please refer to our response in Weakness one for clarification on training compute concerns.

Dear Reviewer,

We sincerely appreciate your effort in reviewing. Could you kindly confirm if our rebuttal addresses your concerns, or if there are any further points we can clarify? We’d be happy to discuss.

Best Regards,

Author

Thank you for taking the time to review our paper.

Here we address the points you mentioned in the weaknesses part.

1. Thank you for the insightful comment. We agree that cognitively complex scenes, particularly those with multiple and potentially overlapping focal points, pose challenges for attention-reduction mechanisms. To explore this, we evaluated Glance2Gaze on counting tasks from the VQA v2 dataset, which exemplify such complexity due to scattered and partially overlapping target objects requiring distributed attention. We compare with LLaVA-1.5-7B under various reduction ratios. As shown in the table below, Glance2Gaze consistently preserves 93.43%–99.20% of its full-retention accuracy, significantly outperforming VisionZip. These results suggest that Glance2Gaze does not concentrate on a single region, but instead maintains broad coverage of all relevant areas in complex visual scenes. We welcome further discussion if you were referring to other types of complex scenes.

   Method	FLOPs	Acc
   LLaVA-1.5-7B	100%	62.4 (100%)
   VisionZip	33%	61.8 (99.04%)
   Glance2Gaze	33%	61.9 (99.20%)
   VisionZip	22%	59.3 (95.03%)
   Glance2Gaze	22%	60.4 (96.79%)
   VisionZip	11%	57.0 (91.35%)
   Glance2Gaze	11%	58.3 (93.43%)

2. In Section 3.3 of the main paper, we report FLOPs and also include CUDA prefilling time in Table 4 as a proxy for latency. We now included a comparison of overall latency and throughout in below table on the TextVQA dataset. Glance2Gaze achieves a 3.15× speedup in overall inference latency, a 6.39× speedup in prefilling time and a 3.135× increase in throughput compared to the vanilla model. These results confirm that our method offers substantial end-to-end efficiency gains, not just in the prefilling phase.

   Method	TFLOPs	Cuda Time (ms)	Latency (ms)	Throughput
   LLaVA-NeXT	53.83	313	475	10.884
   FastV	2.76	170 (1.84 $\times$)	358 (1.33 $\times$)	12.272 (1.128 $\times$)
   SparseVLM	2.76	183 (1.71 $\times$)	379 (1.25 $\times$)	13.233 (1.216 $\times$)
   VisionZip	2.76	57 (5.49 $\times$)	192 (2.47 $\times$)	26.983 (2.479 $\times$)
   Glance2Gaze	2.67	49 (6.39 $\times$)	151 (3.15 $\times$)	34.121 (3.135 $\times$)

3. Thank you for raising this point. While most standard VQA datasets contain well-aligned image-text pairs, we recognize the need for benchmarks that intentionally introduce image-text mismatch while still requiring visual grounding. Despite an extensive search, we were unable to identify publicly available datasets with these characteristics, and we would welcome recommendations for future study. As an alternative, we evaluated POPE, a benchmark for hallucination scenarios where textual descriptions reference non-existent objects. Results in Tables 1~2 in the main paper demonstrate Glance2Gaze's robustness across reduction ratios, significantly outperforming other pruning methods.

4. We appreciate reviewer's attention on effectiveness of Glance Fusion module and learnable queries in Gaze Compression.

   1). In the main paper, we conducted targeted ablations to evaluate the contribution of the Glance Fusion module. As shown in Figure 4, the “Gaze Compression Only” baseline disables Glance Fusion, revealing a notable performance drop discussed in lines 282–288. Additionally, Table 5 compares Glance Fusion as a plug-and-play component against alternative fusion strategies, with detailed analysis in lines 289–296. Both results consistently confirms Glance Fusion's effectiveness.

   2). Accordingly, we implemented a baseline that removes tokens solely based on instruction-to-image attention scores starting from the $r$-th decoder layer. To ensure fairness, we fixed $r=9$ and $p_r=256$, replicating the exponential token retention schedule used in Glance2Gaze. At each layer from $r$ onward, visual tokens were ranked by their attention importance, and the lowest-ranked tokens were pruned according to the predetermined count. As shown in the table below, this attention-based pruning causes substantial performance degradation. Even after fine-tuning (The visual encoder is frozen while all other parameters are fine-tuned for 1 epoch, using identical training configurations as Glance2Gaze), the baseline fails to match Glance2Gaze’s performance, confirming that static pruning based on local attention scores lacks the adaptability required for task-specific compression.

   Method	POPE	SQA	MME	GQA	SEED	MMB	VQA v2	TextVQA	MMVet	LLaVA-B	Avg.
   w/o finetune	83.4	69.1	1775	59.7	54.7	63.0	74.6	55.8	30.6	62.0	96.1%
   w finetune	84.7	69.1	1793	60.8	56.8	63.2	75.7	56.4	31.4	65.2	97.9%
   Glance2Gaze	85.5	70.4	1812	61.5	58.7	64.5	77.6	57.2	32.7	66.4	99.9%

Then here are the answers to the Questions part.

1. While full retention theoretically preserves maximal information, it may introduce redundant or irrelevant features that degrade model performance, an effect similarly observed in early deep networks, where increasing depth did not always yield better results compared to shallower architectures. In contrast, our approach avoids naive compression by employing a structured fusion mechanism (e.g., Glance Fusion) that selectively integrates visual signals aligned with task-relevant semantics.

Dear Reviewer,

We sincerely appreciate your effort in reviewing. Could you kindly confirm if our rebuttal addresses your concerns, or if there are any further points we can clarify? We’d be happy to discuss.

Best Regards,

Author

Thank you for your valuable feedback.

Here we address the points you mentioned in the weaknesses part.

1. We appreciate the reviewer’s attention to the choice of ViT layers and compression sequence.

   1). Our selection of layers {7,13,19,23} was guided by both visual insights and empirical evidence. As illustrated in Figure 1 (left) in the main paper, while ViT layers exhibit varied responses to instruction tokens, adjacent layers yield highly correlated representations. For empirical evidence, we computed average attentional responses of text tokens across 24 ViT layers using the TextVQA and ScienceQA datasets, resulting in the following normalized distribution:

   Layer Range	Values
   1–7	0.0393, 0.0393, 0.0395, 0.0403, 0.0405, 0.0416, 0.0442
   8–17	0.0400, 0.0401, 0.0398, 0.0398, 0.0397, 0.0408, 0.0394, 0.0392, 0.0395, 0.0395
   18–20	0.0406, 0.0432, 0.0412
   21–23	0.0627, 0.0634, 0.0658

   Within each range, we selected the layer exhibiting the highest response correlation to instruction tokens, yielding the set {7,13,19,23}. For further ablation, we tested the second-best layers in each range, {6,9,20,22}. As shown in the table below, both sets significantly outperformed the baseline, but the original selection gave marginally higher gains.

   Method	GQA	VQA v2	SQA	TextVQA	POPE
   LLaVA-1.5-7B	62.0	78.5	66.8	58.2	85.9
   + Glance Fusion $[7, 13, 19, 23]$	64.1 (+2.1)	79.6 (+1.1)	70.6 (+3.8)	59.4 (+1.2)	87.2 (+1.3)
   + Glance Fusion $[6, 9, 20, 22]$	64.1 (+2.1)	79.4 (+0.9)	70.5 (+3.7)	59.4 (+1.2)	86.9 (+1.0)

   2). An exponential decay schedule (the function is ${\frac 2 {p_r}}^{1/(R-r)}$, specific number for each layer are given in Table 1 in the supplementary) was adopted in Gaze Compression to enable stable and progressive compression of visual information. We here additionally evaluated two variant schedules: fast-to-slow decay and slow-to-fast decay. Both resulted in performance degradation, with the fast-to-slow strategy yielding the most significant drop. We hypothesize that aggressive early compression leads to substantial information loss, causing irreversible harm. Results are presented in the table below.

   Strategy	POPE	SQA	MME	GQA	SEED	MMB	VQAv2	TextVQA	MMVet	LLaVA-B	Avg.
   Exponential Decay (Adopted)	83.1	69.1	1722	56.9	53.9	61.7	74.9	55.7	31.4	64.1	95.6%
   Fast->Slow	82.7	68.4	1686	55.4	53.8	59.7	72.6	54.8	30.7	63.3	93.9%
   Slow->Fast	83.0	69.2	1712	56.0	53.2	61.2	73.8	55.2	31.3	63.7	94.9%

2. Thank you for pointing this out. We now included a comparison of overall latency in below table on the TextVQA dataset. Glance2Gaze achieves a 3.15× speedup in overall inference latency and a 6.39× speedup in prefilling time compared to the vanilla model. These results confirm that our method offers substantial end-to-end efficiency gains, not just in the prefilling phase.

   Method	TFLOPs	Cuda Time (ms)	Latency (ms)
   LLaVA-NeXT	53.83	313	475
   FastV	2.76	170 (1.84 $\times$)	358 (1.33 $\times$)
   SparseVLM	2.76	183 (1.71 $\times$)	379 (1.25 $\times$)
   VisionZip	2.76	57 (5.49 $\times$)	192 (2.47 $\times$)
   Glance2Gaze	2.67	49 (6.39 $\times$)	151 (3.15 $\times$)

3. We acknowledge that Glance2Gaze requires model finetuning, unlike training-free approaches. However, the results reported for VisionZip correspond to its fine-tuned variant (as noted in their paper), ensuring a fair comparison. Importantly, despite requiring training, Glance2Gaze significantly reduces overall training cost: as shown in our response to Reviewer CY4r (W1), it achieves a 76.2% reduction in LLaVA-Next-7B training time, operating at only 5% of full-model FLOPs, while retaining 94.8% of the original model’s performance. This exceeds VisionZip’s fine-tuned performance (93.6%) under comparable compression, demonstrating that the modest training overhead yields superior efficiency–performance trade-offs and positions Glance2Gaze as a highly competitive alternative.

4. We thank the reviewer for highlighting the cognitive motivation behind Glance2Gaze. Conceptually, Glance2Gaze is motivated by a well-established cognitive principle, it explicitly separates what to attend to from how to resolve it. This two-stage abstraction enhances spatial reasoning and compositional generalization, contributing to improved performance. Our current goal is to demonstrate the proposed approach's effectiveness through extensive experiments. We acknowledge that a stronger theoretical foundation, e.g., via information-theoretic or attention-based modeling, would further strengthen the work and we consider it an important direction for future research.

Then here are the answers to the Questions part.

1. Please see the answers to Weakness one.

2. We thank the reviewer for raising this point.

   1). As mentioned in lines 200–201 of the main paper, the breakdown is provided in Supplementary A.2. However, we would like to acknowledge that Equation (1) in the Glance Fusion section of the supplementary initially omitted the projection operations. After incorporating all relevant projection layers, we recalculated the overhead introduced by Glance Fusion to be approximately 0.03 TFLOPs. Given that the language model requires around 1.1 TFLOPs to process visual tokens during the prefilling stage, the updated overhead ratio is 0.0272. We will revise both the supplementary and main text to reflect this corrected value for transparency and accuracy.

   2). The primary distinction between the prefilling and decoding (inference) stages of LLMs lies in token processing. During prefilling, all input tokens, including instruction text and image tokens, are processed in parallel. In contrast, decoding omits image token computation, as their hidden states are cached in the KV store. Consequently, the FLOPs associated with image tokens pertain exclusively to the prefilling stage.

Then here are the answers to the Limitations part.

1. We appreciate the reviewer’s valuable suggestion to compare Gaze Compression with attention-based token pruning at a fixed compression ratio. Accordingly, we implemented a baseline that removes tokens solely based on instruction-to-image attention scores starting from the $r$-th decoder layer. To ensure fairness, we fixed $r=9$ and $p_r=256$, replicating the exponential token retention schedule used in Glance2Gaze. At each layer from $r$ onward, visual tokens were ranked by their attention importance, and the lowest-ranked tokens were pruned according to the predetermined count. As shown in the table below, this attention-based pruning causes substantial performance degradation. Even after fine-tuning (The visual encoder is frozen while all other parameters are fine-tuned for 1 epoch, using identical training configurations as Glance2Gaze), the baseline fails to match Glance2Gaze’s performance, confirming that static pruning based on local attention scores lacks the adaptability required for task-specific compression.

   Method	POPE	SQA	MME	GQA	SEED	MMB	VQA v2	TextVQA	MMVet	LLaVA-B	Avg.
   w/o finetune	83.4	69.1	1775	59.7	54.7	63.0	74.6	55.8	30.6	62.0	96.1%
   w finetune	84.7	69.1	1793	60.8	56.8	63.2	75.7	56.4	31.4	65.2	97.9%
   Glance2Gaze	85.5	70.4	1812	61.5	58.7	64.5	77.6	57.2	32.7	66.4	99.9%

Dear Reviewer,

We sincerely appreciate your effort in reviewing. Could you kindly confirm if our rebuttal addresses your concerns, or if there are any further points we can clarify? We’d be happy to discuss.

Best Regards,

Author