Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models

We sincerely thank the reviewer for the detailed and helpful review. We apologize for any confusion caused by the incomplete results in the original submission. Your comments have been very valuable, and we provide our detailed responses below. We would also appreciate any further suggestions or feedback.

1. Quantitative Evaluation for Attention Quality

Thank you for raising this important point. We agree that intrinsic, quantitative evidence is necessary to directly support our claims regarding attention quality.

To this end, we analyzed visual attention by measuring how much attention is assigned to semantically relevant image regions during the decoding process, using 5,000 randomly sampled images from the MSCOCO 2014 validation set. Specifically, for each generated token corresponding to a ground truth object in the caption, we calculated the total attention score allocated to image tokens within the region of that object.

To identify these regions, we used an open-vocabulary segmentation model (Grounded SAM2 [1]) to generate binary masks for all ground truth objects. During caption generation, for each object word token, we measured the proportion of visual attention focused on the corresponding object region out of the total attention score across all image tokens.

The table below summarizes the results across three methods:

These results indicate that our method achieves better alignment between generated text and relevant visual regions compared to the baseline and naive attention scaling. This suggests that our model's visual attention is less noisy and more focused on semantically meaningful parts of the image during captioning.

[1] Ren, T., Shen, S. Grounded SAM 2: Ground and Track Anything in Videos. IDEA-Research, GitHub, 2024.

---

2. Computational Efficiency Comparison

Thank you for your comment. We agree that empirical evidence is important to support our efficiency claims. A detailed comparison of runtime and memory overhead—including per-token generation latency and storage requirements—is provided in our response to Reviewer JgJt. To avoid redundancy, we kindly refer you to that section for a full breakdown. In brief, our method introduces only minimal overhead compared to the baseline, while remaining significantly more efficient than prior approaches.

---

3. More Ablations for Parameters

Thank you for raising this important concern regarding hyperparameter sensitivity. As mentioned in the main paper, we initially conducted ablation studies on hyperparameters using the LLaVA-1.5 (7B) model on the IIW-400 dataset. The results are presented in Tables 6–9 of the supplementary material.

To further strengthen our analysis and address the reviewer’s concern, we extended these ablations to additional models and datasets. Specifically, we conducted experiments using LLaVA-NeXT (7B) and Qwen2-VL (7B) on the DOCCI dataset. We randomly sampled 500 images and evaluated the quality of the generated captions using the CLAIR score. The results are summarized below.

LLaVA-NeXT (7B)

Baseline CLAIR score: 62.49

Ours (Layer 20, $\tau$=4, $\alpha$=1.1, $\beta$=0.1): 66.99

Qwen2-VL (7B)

Baseline CLAIR score: 79.22

Ours (Layer 18, $\tau$=3, $\alpha$=1.1, $\beta$=0.1): 80.64

Across all models (LLaVA-1.5, LLaVA-NeXT, Qwen2-VL), we observe similar trends regarding the optimal range for the layer, $\alpha$, $\beta$, and $\tau$ parameters—typically favoring mid-to-late transformer layers and slightly scaled values. We also note that our method is training-free and imposes minimal additional computational cost, making it practical and efficient to perform lightweight hyperparameter searches in real-world applications.

We sincerely thank the reviewer for the thoughtful feedback and insightful question. Your comments greatly helped us refine and clarify the paper. Please find our detailed response below—we’re happy to address any further concerns.

1. Analysis of the “noisy” scores

We understand the reviewer’s concern regarding whether the increased attention diversity in our method may be attributed to “noisy” scores. To investigate this, we conducted an analysis of attention “noisiness” by measuring the proportion of attention assigned to semantically relevant image regions during decoding, using 5,000 randomly sampled images from the MSCOCO 2014 validation set.

Specifically, for each generated token corresponding to a ground truth object word in the caption, we computed the proportion of total attention allocated to image tokens within the region of that object, relative to the total attention across all image tokens. These object regions were identified using binary masks obtained from an open-vocabulary segmentation model, Grounded SAM2 [1], applied to each object in the image.

The table below compares the average attention allocated to relevant image regions across three methods: Baseline, Ours, and Naive Attention Scaling. A higher proportion indicates that the model is focusing more accurately on the relevant visual regions, suggesting lower attention noise.

These results indicate that our method assigns more focused attention to semantically meaningful image areas compared to the baseline and naive attention scaling, suggesting that our method results in less noisy visual attention.

Furthermore, in Fig. 13(a), we report the caption similarity score, which measures the semantic similarity between sentences within a generated caption. Captions generated by our method show greater diversity in sentence content, reinforcing that the diversity is not due to randomness or noise in attention, but rather stems from meaningful differentiation in visual grounding and language generation.

[1] Ren et al. Grounded SAM 2: Ground and Track Anything in Videos. IDEA-Research, GitHub, 2024.

---

2. More Ablations for Parameters

We appreciate the reviewer’s suggestion regarding additional ablation studies on different models and datasets. To address this point, we conducted additional ablation studies on LLaVA-NeXT (7B) and Qwen2-VL (7B) using the DOCCI dataset, analogous to Tables 6–9 in the paper. To avoid redundancy, we kindly refer you to our response to Reviewer 4gUZ, where detailed results and observations are provided. In brief, we observed consistent trends across models, with performance generally favoring mid-to-late layers and modest parameter scaling. Since our method is training-free and efficient, such tuning remains practical.

3. Clarification Regarding “Noisy” Attention Scores

We sincerely apologize for any confusion caused by our use of the term “noisy” to describe certain attention scores. As discussed in [2], background tokens can carry important global context and are often beneficial in vision transformers.

Our intent was not to imply that such attention is inherently harmful. Rather, our goal is to enhance attention to semantically relevant visual regions during decoding—not to suppress background tokens altogether. In fact, we agree that background tokens can carry useful information, and our method does not explicitly penalize them.

However, we found that naive attention scaling may unintentionally amplify attention to globally dominant tokens (including background regions), which can overwhelm local, task-relevant signals. This may explain why captions generated with naive attention scaling often exhibit high precision but reduced recall—they tend to overemphasize prominent visual cues while neglecting finer details.

We sincerely thank the reviewer for pointing this out and will revise the manuscript to replace “noisy” with more accurate wording.

[2] Darcet et al. Vision Transformers Need Registers. ICLR’24.

4. Attention weight trends for text and image tokens as context length increases

Fig. 5 does not account for the share of total tokens.

Thank you for the insightful comment. As noted, Fig. 5 did not normalize for the total number of tokens. To address this, we analyzed the average attention per token and found that attention to image tokens decreases disproportionately faster than to text tokens as context length increases.

5. Clarification on the meaning of “repeating” in evaluation

Thank you for the question. The “repeating” refers to resampling 500 instances from the MSCOCO validation set five times, following prior work [3]. Generation was deterministic with temperature set to 0.

Liu et al. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. ECCV’24. [3]

1. Concerns Regarding Performance Trade-off

Compared with baselines, the proposed method significantly improves the recall, but the precision is hurt. For example, as shown in Table 2, the precision is ~3% lower than PAI. Similarly, in Figure 6, the human evaluation suggests a lower precision compared with "Naive" (PAI). This may lead to concerns on hallucination and misleading information in the captions. Ideally, the method should be able to adjust its hyperparameters to achieve a higher recall without hurting the precision.

We appreciate the reviewer’s concern regarding the potential trade-off between recall and precision in our method. However, we would like to clarify that, as shown in Table 2, our method achieves improvements in both precision and recall compared to the baseline. In contrast, other prior methods, including "Naive" (PAI), tend to improve precision at the cost of recall.

In particular, while PAI demonstrates a higher precision than our method, it substantially reduces recall, which may not be acceptable for applications where completeness of information is critical. This is especially relevant in real-world scenarios such as medical image reporting or automated content generation, where omitting important visual elements may lead to misleading or insufficient descriptions. In such cases, recall can be more crucial than precision, as overly high precision may inadvertently filter out meaningful or necessary content.

---

2. Evaluation on other hallucination benchmark

Thank you for this valuable suggestion. While our primary focus is on enhancing detailed image captioning, we also performed additional experiments on the POPE hallucination benchmark [2],

Inspired by the evaluation setup proposed in [1], where caption-based reasoning improves MLLM performance on general multimodal tasks, we adopt a similar strategy. Specifically, we first generate a caption for the image and then use it as part of the input to answer the question.

We evaluated Qwen2-VL (7B) using three approaches: the baseline model, our proposed method, and naive attention scaling (PAI). For the naive method, we identified the optimal hyperparameter ($\alpha$ = 0.2) before evaluation.

The table below summarizes the accuracy on the POPE benchmark:

Our method shows an improvement over the baseline, while the naive attention scaling slightly reduces accuracy. Additionally, we evaluated the instruction-following behavior, measured as the proportion of responses in which the model correctly generates an output that includes both a caption and an answer.

These results indicate that naive attention scaling may reduce the model's sensitivity to the input prompt, whereas our method retains alignment with instruction while improving grounding accuracy.

---

3. Concerns Regarding the Choice of Hyperparameters

We appreciate the reviewer’s thoughtful concern regarding the generalizability of hyperparameter settings across different models. To address this, we conducted additional ablations on LLaVA-NeXT (7B) and Qwen2-VL (7B) using the DOCCI dataset. For detailed results, please refer to our response to Reviewer 4gUZ. Briefly, across all models (LLaVA-1.5, LLaVA-NeXT, Qwen2-VL), we observe similar trends regarding the optimal range for the layer, $\alpha$, $\beta$, and $\tau$ parameters—typically favoring mid-to-late transformer layers and slightly scaled values. We also note that our method is training-free and imposes minimal additional computational cost, making it practical and efficient to perform lightweight hyperparameter searches in real-world applications.

---

4. Computational Efficiency Comparison

Thank you for this valuable suggestion. A detailed comparison of runtime and memory overhead—including per-token generation latency and storage requirements—is provided in our response to Reviewer JgJt. To avoid redundancy, we kindly refer you to that section for a full breakdown. In brief, our method introduces only minimal overhead compared to the baseline, while remaining significantly more efficient than prior approaches.

We sincerely thank the reviewer for taking the time and effort to evaluate our paper. We truly appreciate your insightful comments. Please find our detailed response to your comment below. If you have any further feedback, we would be grateful to hear it.

1. Efficiency Comparisons

Since the paper involves additional attention amplification procedures, efficiency comparisons can be included. e.g., token generation speed.

Thank you for your valuable suggestion. While our method involves attention amplification procedures, these introduce only minimal overhead compared to the original decoding process. In contrast, many previously proposed approaches require additional decoding passes, resulting in significant computational cost.

To provide a clearer comparison, we measured the token generation time (i.e., generation time per output token) across various methods. The following table presents the average generation time per token (in milliseconds) using an RTX8000 GPU:

As shown, our method performs similarly to the baseline, whereas other methods introduce 2x to 10x slower generation speeds. This demonstrates that our approach achieves efficient generation with minimal computational overhead.

Regarding memory usage: our method only requires storing the head-wise averaged attention scores for image tokens at each layer from the previous decoding step. For example, in the case of LLaVA-1.5 (7B), this amounts to:

32 (layers) × 576 (image tokens) × 2 bytes (float16) < 40 KB

This overhead is negligible in modern hardware setups.