AdaptVis: Spatial Understanding in Vision-Language Models Requires Adaptive Attention

Thank you for your constructive review.

Writing

Thanks for pointing it out. We have thoroughly revised and proofread the manuscript to enhance clarity and ensure proper definition of all terminology. Our paper focuses on the mechanism interpretability of spatial reasoning. It opens the black box to see why VLMs fail in simple spatial reasoning and how the attention of VLMs behaves under spatial reasoning questions, so we believe mechanism interpretability analysis is of the same importance as the method and provide various analyses in Section 3. We made it clear in the revision.

Suggestion of creating an adaptive algorithm to set thresholds for specific scenarios.

We further explored additional metrics in Appendix E to automatically distinguish the distributions and uncover their underlying characteristics during our preliminary experiments. Specifically, we examined entropy and skewness. Entropy was selected based on the hypothesis that parameter differences may stem from the familiarity of attention patterns in real images, which are generally correct, versus synthetic images, where these patterns tend to be incorrect. We posit that the model can express "familiarity" through certain metrics derived from the attention scores. For example, we hypothesize that the entropy of attention will be lower when the model encounters familiar cases.

Potential Overfitting

First, the hyperparameter only requires a small number of instances and is more like a calibration. To indicate this, we present the results with models using the same hyperparameter in Table 4, and we can still see consistent improvement. Using the same prompt complexity (all with 4 options), same alpha12 (temperature), and beta (threshold) parameters showed consistent improvements across different benchmarks.

Second, we provide additional results on generic question-answering settings such as GQA, VQAv2, and POPE, as added in Appendix G.7. Our method consistently outperforms the baseline across all benchmarks but with much less gain. It shows that adjusting attention is important and has generalization capability in the general Question Answering setting, but may require different adaptation strategies, which can be interesting future work.

Thank you for the constructive review.

Only focuses on spatial reasoning

We appreciate the reviewer’s appreciation and trust in our mechanism interpretability method for generic problems. However, we respectfully disagree that the focus of spatial reasoning is a limitation. It is well-established in the field that mechanism interpretability studies must necessarily focus on specific capabilities or behaviors, as large models exhibit distinct mechanisms across different tasks, different domains, and even different input formats. This targeted approach is not a limitation but rather a methodological necessity, as evidenced by previous successful mechanism interpretability work. We specifically chose spatial reasoning because it represents a critical bottleneck in VLMs' visual understanding capabilities, and our paper presents the first-ever mechanism interpretability work of this important problem. The specificity of our analysis allows us to make precise, actionable discoveries about how VLMs process simple spatial reasoning.

Additional Datasets

We provide additional results on generic question-answering settings such as GQA, VQAv2, and POPE. Our method consistently outperforms the baseline across all benchmarks but with much less gain. It shows that adjusting attention is important and has generalization capability in the general Question Answering setting, but may require different adaptation strategies, which can be interesting future work.

Usage of Hyperparameters

First, the hyperparameter only requires a small number of instances and is more like a calibration. To indicate this, we present the results with models using the same hyperparameter in Table 4, and we can still see consistent improvement. Using the same prompt complexity (all with 4 options), same alpha12 (temperature), and beta (threshold) parameters showed consistent improvements across different benchmarks.

Second, we further explored additional metrics in Appendix E to automatically distinguish the distributions and uncover their underlying characteristics during our preliminary experiments. Specifically, we examined entropy and skewness. Entropy was selected based on the hypothesis that parameter differences may stem from the familiarity of attention patterns in real images, which are generally correct, versus synthetic images, where these patterns tend to be incorrect. We posit that the model can express "familiarity" through certain metrics derived from the attention scores. For example, we hypothesize that the entropy of attention will be lower when the model encounters familiar cases.

Thank you for your constructive review.

Clarity of Section 3.2 and Section 4.1

In our preliminary experiments, we conducted extensive analyses of attention patterns and their connection to factual errors. We did not include them in the original paper since our observation that "models primarily process information in intermediate layers" aligns with previous findings in [1] and [2]. Additionally, our conclusion that some VLM failures can be attributed to incorrect attention is consistent with the findings in [3]. Due to page limitations, we only include results that show more novel observations in the paper.

The revised version will have additional, comprehensive results in appendices, including an analysis of image attention score values across layers on all subsets of WhatsUp and visualizations of attention areas.

References:

1. Overthinking the Truth: Understanding How Language Models Process False Demonstrations, Danny Halawi et al., 2023.
2. Dissecting Recall of Factual Associations in Auto-Regressive Language Models, Mor Geva et al., 2024.
3. Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training, David Wan et al., 2024.

Figure 7

Thank you for your suggestion. We have clarified the caption to improve its clarity. The average confidence score is calculated as the mean of the model's confidence across each type of question with different ground-truth labels, referred to here as "golden" labels. For COCO_two, we made a typo in Figure 7 (left panel), where COCO2 and VG2 were reversed. This typo has been corrected in the latest version of our work. Regarding the predictions, the pattern aligns closely with the confidence scores, as shown in Figure 10. We observe that the trend in confidence scores corresponds closely to the accuracy scores for each type of ground-truth label.

About ScalingVis

In Lines 223–225, the experimental setup involves adding a constant coefficient equally to all image tokens, corresponding to an addition operation in the logit space (equivalent to a multiplication operation in the probability space). In contrast, Scaling Vis uses a multiplication operation directly in the logit space, making it an adaptive approach. Regarding "overfitting," we have shown that our hyperparameter performs well in general settings, as demonstrated in Table 4. As for why sharpening versus smoothing the image attention supports the two datasets (synthetic vs. real), we hypothesize that this is due to differences in their label distributions. Synthetic data contains more unfamiliar relationships, as discussed in Section 4.2, which benefits from a smoother image attention distribution, as observed in Figure 10. In contrast, real images feature more familiar relationships that require a sharper image attention distribution, as shown in Figure 10 as well.

About Baselines

We have added the baseline introduction in the latest version of our paper.

Experimental results

• We especially focus on the mechanism interpretability of spatial reasoning and open the black box to see how attention behaves under spatial reasoning questions, since it is well-established in the field that mechanism interpretability studies must necessarily focus on specific capabilities or behaviors, as large models exhibit distinct mechanisms across different tasks, different domains, and even different input formats. This targeted approach is not a limitation but rather a methodological necessity, as evidenced by previous successful mechanism interpretability work. We specifically chose spatial reasoning because it represents a critical bottleneck in VLMs' visual understanding capabilities, and our paper presents the first-ever mechanism interpretability work of this important problem. The specificity of our analysis allows us to make precise, actionable discoveries about how VLMs process simple spatial reasoning.
• Spatial reasoning especially requires attention on the relative region to reflect the geometric structures, which is a relatively simple and focused attention adaptation, while generic images may have much more complicated attention working mechanisms.
• We provide additional results on generic question-answering settings such as GQA, VQAv2, and POPE. Our method consistently outperforms the baseline across all benchmarks but with much less gain. It shows that adjusting attention is important and has generalization capability in the general Question Answering setting, but may require different adaptation strategies, which can be interesting future work.

We thank the reviewers for appreciating the idea of modifying the attention.

The analyses in Sections 3 and 4

• In our preliminary experiments, we conducted extensive analyses of attention patterns and their connection to factual errors. We did not include them in the original paper since our observation that "models primarily process information in intermediate layers" aligns with previous findings in [1] and [2]. Additionally, our conclusion that some VLM failures can be attributed to incorrect attention is consistent with the findings in [3]. Due to page limitations, we only include results that show more novel observations in the paper.
• The revised version will include comprehensive results in appendices, including an analysis of image attention score values across layers on all subsets of WhatsUp and visualizations of attention areas.

References

1. Overthinking the Truth: Understanding How Language Models Process False Demonstrations, Danny Halawi et al., 2023.
2. Dissecting Recall of Factual Associations in Auto-Regressive Language Models, Mor Geva et al., 2024.
3. Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training, David Wan et al., 2024.

Reliability of Attention Modification

• The key idea is to be more focused if the model is confident and widen the focus if the model is unsure. In this case, if the model is not confident, we assume the model should not stick to the original wrong prediction, where “further reducing confidence” is what we want.
• We especially focus on spatial reasoning and open the black box to see how the attention of VLMs behaves under spatial reasoning questions, since it is well-established in the field that mechanism interpretability studies must necessarily focus on specific capabilities or behaviors, as large models exhibit distinct mechanisms across different tasks, different domains, and even different input formats. This targeted approach is not a limitation but rather a methodological necessity, as evidenced by previous successful mechanism interpretability work. We specifically chose spatial reasoning because it represents a critical bottleneck in VLMs' visual understanding capabilities, and our paper presents the first-ever mechanism interpretability work of this important problem. The specificity of our analysis allows us to make precise, actionable discoveries about how VLMs process simple spatial reasoning.
• Spatial reasoning especially requires attention on the relative region to reflect the geometric structures, which is a relatively simple and focused attention adaptation, while generic images may have much more complicated attention working mechanisms.
• We provide additional results on generic question-answering settings such as GQA, VQAv2, and POPE. Our method consistently outperforms the baseline across all benchmarks, but with much less gain. It shows that adjusting attention is important and has generalization capability in the general Question Answering setting, but may require different adaptation strategies, which can be interesting future work.

Undefined Terms

Thanks for pointing it out. We have thoroughly revised and proofread the manuscript to enhance clarity and ensure proper definition of all terminology.

Withdrawal reversed temporarily.