Why context matters in VQA & Reasoning: Semantic interventions for VLM input modalities

Thank you for your comments and for raising these important points. We appreciate the opportunity to clarify our findings and address your concerns.

1. Dataset size.

Regarding the dataset size,

• Evaluation purpose. We emphasize that our dataset is only used for evaluation, not for training or fine-tuning models. Its purpose is to analyze pretrained models’ behavior when faced with diverse input configurations.
• TinyBenchmark. While the SI-VQA dataset we designed remains small, prior recent research [1] demonstrates that smaller datasets can still yield meaningful results when the quality of instances is high. Each instance of the SI-VQA Dataset underwent multiple rounds of refinement, ensuring that context descriptions serve only as background information without providing enough detail to answer the question independently.
• Low Variance. Despite the small size, the dataset exhibits low variance in model performance and uncertainty, as demonstrated in the Appendix H. Increasing the number of instances would not alter the conclusions drawn, as they hinge on the modality configurations rather than the dataset size. Instead, rebalancing the role of modalities, such as prioritizing text and introducing spurious visual information, would yield new insights—a direction we are actively exploring.
• Scalability and Community Engagement. To ensure data quality and consistency, we intentionally limited the dataset size to 100 well-curated instances. However, we have also developed a ready-to-use interactive interface that allows the community to generate additional instances with new images and textual descriptions. This ensures that our SI-VQA dataset is extensible and can grow with greater diversity and community input. See Appendix E for details about the Interface and instructions to create new data instances and the link to the code to try it out: https://gitlab.com/dekfsx1/isi-vlm.

[1] tinyBenchmarks: evaluating LLMs with fewer examples

In the revised version of the paper, we integrated this argumentation in the Limitations and Future Directions paragraph in the Discussion.

2. The authors are suggested to test larger model sizes, such as 13B models - LLaVA-1.5-vicuna-13B.

We appreciate the reviewer’s suggestion to test larger model sizes, such as 13B, but our study is focused on observing performance changes across different strategies rather than achieving the highest possible accuracy. Our chosen 7B model aligns with the difficulty of our VQA dataset and provides an optimal balance for meaningful performance variation. In addition, our ablation study in Appendix Section F shows minimal information loss with quantization, supporting the suitability of the 7B model size for our analysis. Using a larger model would obscure these nuanced differences and shift focus away from our core objective, thus we only recommended it for further research.

We have added a sentence in section 4.1: “We restrict our analysis to 7B models because this number of parameters aligns with the difficulty of our VQA dataset and provides an optimal balance for meaningful performance variation.”

3. It seems like there is a strong connection between this work and several well-studied problems such as modality bias (language prior) in VQA [1][2], and visual commonsense reasoning (VCR) [3].

We thank the reviewer for bringing these three research papers to our attention. While these works focus on the role of modalities in vision-language models, their direct relevance to our study is limited due to differences in objectives and scope. Before elaborating further, we would like to clarify what our paper is not:

1. We are not proposing a new large-scale multimodal benchmark aimed at achieving state-of-the-art results in multimodal learning.
2. We are not attempting to resolve modality bias.

With these distinctions in mind, we would like to explain how the cited works relate to our paper:

[1] "On Modality Bias Recognition and Reduction"

This paper explores modality bias in VLMs and proposes a fine-tuning strategy using a de-biasing loss function. While we acknowledge the importance of addressing modality bias, our objective is not to mitigate these biases directly. Instead, we aim to study and understand such biases by experimenting with modality configurations, specifically in the context of the SI-VQA dataset. Using pre-trained VLMs that come with inherent inductive biases, we test two methods to rebalance modality importance (“de-biasing”) and report diverse and, at times, surprising conclusions. Our work complements [1] by exploring the interplay of modalities rather than focusing on reducing modality bias.

[2] "Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning"

This paper investigates the existence of a modality gap in the embedding space under various initializations and linear activations, offering an analysis of how the gap impacts downstream performance and fairness. While we find this work highly relevant to understanding modality roles, our focus is different: we examine the effects of modality interplay on VLM behavior (e.g., performance, uncertainty) and characteristics (e.g., attention distribution) in specific SI-VQA configurations.
That said, the insights from [2] inspire exciting future research directions for our work. For instance, studying how the modality gap and cone effect evolve across the seven SI-VQA configurations could deepen our understanding of the interaction between embedding geometry and modality configurations. In addition, applying the techniques from [2] to modify the modality gap could serve as valuable ablation studies, helping assess how altering this gap influences VLM performance on SI-VQA tasks. We greatly appreciate the reviewer for highlighting this connection and pointing us toward opportunities for future research.

[3] "From Recognition to Cognition: Visual Commonsense Reasoning"

This paper introduces a large-scale dataset focused on visual commonsense reasoning, addressing biases such as answer-only biases through adversarial matching. Their objective is to create a robust dataset with questions, answers, and rationales, primarily targeting multi-choice Q&A settings affected by artifact annotations. In contrast, our study does not aim to construct a large-scale VQA dataset that is robust to such biases. Instead, we investigate modality biases by manipulating modality configurations in the SI-VQA dataset.
Moreover, in our dataset, the model consistently attends to the question due to the task design, and we opted for close-ended questions over multiple-choice formats to minimize biases observed in early project experiments. Our objectives, therefore, differ significantly from [3], focusing on analyzing biases through controlled modality interplay rather than eliminating them in large-scale datasets.

We are grateful to the reviewer for these suggestions, as they not only enrich our understanding of the field but also inspire directions for extending our work in future research. Thank you:)

Regarding the dataset size,

• Evaluation purpose. We emphasize that our dataset is only used for evaluation, not for training or fine-tuning models. Its purpose is to analyze pretrained models' behavior when faced with diverse input configurations. 

• TinyBenchmark. While the SI-VQA dataset we designed remains small, prior recent research [1] demonstrates that smaller datasets can still yield meaningful results when the quality of instances is high. Each instance of the SI-VQA Dataset underwent multiple rounds of refinement, ensuring that context descriptions serve only as background information without providing enough detail to answer the question independently.

• Low Variance. Despite the small size, the dataset exhibits low variance in model performance and uncertainty, as demonstrated in the Appendix H. Increasing the number of instances would not alter the conclusions drawn, as they hinge on the modality configurations rather than the dataset size. Instead, rebalancing the role of modalities, such as prioritizing text and introducing spurious visual information, would yield new insights---a direction we are actively exploring.

• Scalability and Community Engagement. To ensure data quality and consistency, we intentionally limited the dataset size to 100 well-curated instances. However, we have also developed a ready-to-use interactive interface that allows the community to generate additional instances with new images and textual descriptions. This ensures that our SI-VQA dataset is extensible and can grow with greater diversity and community input. See Appendix E for details about the Interface and instructions to create new data instances and the link to the code to try it out: https://gitlab.com/dekfsx1/isi-vlm.

[1] tinyBenchmarks: evaluating LLMs with fewer examples

In summary, we respectfully argue that our work provides new insights into modality roles in VLMs and addresses a unique, underexplored aspect of multimodal research. We thank the reviewer for the opportunity to clarify our contributions and the broader context of our work.

We have modified the Introduction of the revised version of the paper to clarify the scope of the paper, novelties, and contributions. In the Discussion section, we elaborated more on the arguments about the reduced size. of the dataset

Questions: Are there any other interesting conclusions or exploratory directions for uncovering the importance of multimodal complementarity?

Other conclusion: Our findings suggest that attention to a modality is not necessarily determined by the number of tokens associated with it. For instance, despite LLaVA-Next having approximately six times more image tokens, its attention toward those tokens remains comparable to models with fewer tokens -- like LLaVA-1.5.

Exploratory Direction: A promising avenue for further research is investigating tasks where both modalities provide semantically distinct yet equally essential information. By analyzing scenarios where successful task completion (e.g., answering a question) requires complementary inputs from both modalities, we can deepen our understanding of how models integrate and prioritize multimodal information.

Thank you for your comments and for raising these important points. We appreciate the opportunity to clarify our findings and address your concerns.

1. The dataset contains only 100 samples, and the conclusions drawn lack novelty

We thank the reviewer for their feedback. We acknowledge that the importance of multimodal complementarity is a well-established concept in the field. However, we believe our work brings a unique perspective and addresses gaps in the existing literature.

A Dataset for Model Behavior Analysis - Not A New Large-scale VLM Benchmark

Our dataset and methodology are not designed to benchmark large vision-language models (VLMs) but to probe their behavior in specific, controlled multimodal scenarios. This unique focus on modality roles and interactions represents an important step toward understanding the limits and strengths of VLMs.

Unique modality configurations

As discussed in the Related Work section, prior studies have indeed explored the role of modalities. However, these works often suffer from limitations such as data leakage (e.g., MMMU) or lack of complexity in the multimodal interaction (e.g., MMStar). For instance, MMStar involves a straightforward 4-option multi-class classification task without reasoning, minimal accompanying textual descriptions, and OCR-based dependencies. Additionally, the images in these datasets are not comparable or designed for controlled multimodal analysis. In addition, existing large-scale datasets often suffer from incomplete data, with missing ground-truth explanations (i.e., rationale) for vision-language model (VLM) answers. Moreover, images are not always strictly natural, with some containing pre-existing OCR data (text annotations) that introduce noise. Questions within these datasets vary widely in complexity, leading to inconsistency. In contrast, our curated dataset addresses these issues by featuring purely natural images across diverse domains, consistently complex questions, independent text descriptions, and no pre-existing annotations. This careful construction avoids confounding variables that could skew our results. Shaping unique modality configurations (or scenarios), we can analyze model performance, uncertainty, and attention mechanisms together, providing unprecedented insights into the interaction of modalities in a reasoning context.

Contribution to the Advancement of Vision Language Model Interpretability & Robustness

Understanding how modalities impact VLMs by exploring complex and specific scenarios is crucial as multimodal models become increasingly opaque and diverse. Our work demonstrates how task-specific modality configurations can reveal heterogeneous effects on model performance and uncertainty. This approach not only advances our understanding of VLMs but also highlights areas where these models require improvement to achieve robustness in diverse multimodal tasks.

3. The work lacks comparisons with other current datasets. Even though each sample is well-annotated ..., there are no comparisons between the proposed dataset and state-of-the-art (SOA) datasets regarding its advantages in Image-dependent Answers and Content Domain Diversity.

Given the highly specific setup of our dataset, which is unique in its design, making direct comparisons with state-of-the-art datasets is multifaceted. The SI-VQA dataset offers distinct advantages, particularly the unique combinations of image and text modalities.  

Regarding content domain diversity, we have sourced images from a wide range of domains, including geography, biology, household items, and sports, drawing from datasets such as MMMU and MMStar. This ensures that SI-VQA maintains a comparable diversity of topics to existing SOTA datasets.  

In terms of image-dependent answers, SI-VQA follows the example of MMStar by avoiding data leakage often found in datasets like ScienceQA, AI2D, MMMU, and SEED [3], and annotation artifacts where models can sometimes answer correctly without even considering the question [1]. However, unlike MMStar, which uses multiple-choice questions, we chose a binary question format (Yes/No) to eliminate issues related to distractor quality and random guessing, ensuring more precise evaluations.  

The complexity of questions in SI-VQA strikes a balance: it requires understanding object significance and simple interactions in the image without reaching the higher complexity levels of datasets like Visual Commonsense Reasoning [1] or Scene Graph VQA [2].  

Additionally, the ISI tool, partially used to design the SI-VQA dataset, provides exceptional flexibility to introduce new perturbations and explore the complementarity of text and image modalities. This capability enhances the complexity of comparing SI-VQA dataset to SOTA VQA and reasoning datasets in the future.

In our revised version of the paper, we have extended Section 3.1 to account for a thorough comparison of SI-VQA to state-of-the-art datasets regarding its advantages in Image-dependent Answers and Content Domain Diversity and explanations for the design choices we made.

[1] From Recognition to Cognition: Visual Commonsense Reasoning  

[2] GVQA: Learning to Answer Questions about Graphs with Visualizations via Knowledge Base  

[3] Are We on the Right Way for Evaluating Large Vision-Language Models?

4. The work lacks supporting evidence for its claims. Regarding the claim that image text annotations have minimal impact on accuracy, or even decrease accuracy, the authors list some potential reasons for this, e.g., VLMs may already extract relevant information from images. It would be helpful to provide some qualitative or quantitative results to further support these explanations.

The paper attributes the limited benefit of annotations for the models to information redundancy. We conduct a complementary analysis to demonstrate that redundant information can be either ineffective or even harmful to VLMs: the first ablation study introduces redundant textual information about the image. Our findings show that such redundant descriptions do not enhance model performance, reinforcing the claim that annotations do not improve predictions. As a matter of fact, when models are overloaded with redundant information, their performance may even decline, highlighting the uselessness or potentially detrimental impact of annotations or redundant textual descriptions.

Given the ISI tool, we already planned to extend the SI-VQA dataset with an 8th-modality configuration where the image annotations are not guiding the model toward the correct answer, but rather misleading it. As this new configuration was first planned to observe how VLMs can manage contradicting information within the visual input, this is also an opportunity to bring additional quantitative results to show if "negative" annotations are accounted for, i.e., whether non-redundant OCR plays a significant role, as opposed to redundant "positive" annotations.

5. The work lacks formal definitions of certain terms. The term "modality relevance" is first mentioned in the abstract. However, there is no formal definition provided for it.

We acknowledge the confusion of that term and modified the abstract to change it to a well-defined concept that captures what we meant here: "attention attribution to modality"

Thank you for your comments and for raising these important points. We appreciate the opportunity to clarify our findings and address your concerns.

1. There are some concerns about the dataset. The proposed dataset contains only 100 samples, which is quite limited in this domain.

Regarding the dataset size,

• Evaluation purpose. We emphasize that our dataset is only used for evaluation, not for training or fine-tuning models. Its purpose is to analyze pretrained models' behavior when faced with diverse input configurations. 

• TinyBenchmark. While the SI-VQA dataset we designed remains small, prior recent research [1] demonstrates that smaller datasets can still yield meaningful results when the quality of instances is high. Each instance of the SI-VQA Dataset underwent multiple rounds of refinement, ensuring that context descriptions serve only as background information without providing enough detail to answer the question independently.

• Low Variance. Despite the small size, the dataset exhibits low variance in model performance and uncertainty, as demonstrated in the Appendix H. Increasing the number of instances would not alter the conclusions drawn, as they hinge on the modality configurations rather than the dataset size. Instead, rebalancing the role of modalities, such as prioritizing text and introducing spurious visual information, would yield new insights---a direction we are actively exploring.

• Scalability and Community Engagement. To ensure data quality and consistency, we intentionally limited the dataset size to 100 well-curated instances. However, we have also developed a ready-to-use interactive interface that allows the community to generate additional instances with new images and textual descriptions. This ensures that our SI-VQA dataset is extensible and can grow with greater diversity and community input. See Appendix E for details about the Interface and instructions to create new data instances and the link to the code to try it out: https://gitlab.com/dekfsx1/isi-vlm.

[1] tinyBenchmarks: evaluating LLMs with fewer examples

In the revised version of the paper, we integrated this argumentation in the Limitations and Future Directions paragraph in Discussion.

2.a. Answer distribution. The answers are limited to "Yes" or "No."

Evaluating a large language model (LLM) with binary questions provides clear, unambiguous insights into its performance by simplifying the evaluation to a straightforward correct/incorrect framework. This reduces complexity compared to multiple-choice questions, where distractor quality and guessing can skew results, making binary questions ideal for precise assessments.

To further clarify our choice for the binary setting in this VQA and Reasoning task, we have added a justification in the revised version of the paper, Section 3.1.

2.b. Moreover, the paper does not specify the distribution of "Yes" versus "No" answers in the dataset. This leads to the following two concerns: a. Model Bias, b. Impact of Interventions

The paper specifies the distribution of "Yes" and "No" -- See Appendix H.1. Figure 9, Confusion matrices and accuracy values for all model architectures and configurations, shows a well-balanced answer distribution which addresses the reviewer's concern about model bias and impact intervention true effect measurement.

Questions:

1. It appears that image text annotations have little effect on some of the model's metrics; for example, the results of Q+I_A+C_+ in Figure 2a are not optimal. Could the authors analyze the reason for this phenomenon?

We appreciate the reviewer's question regarding the limited effect of text annotations on the accuracy metrics.

Our observations indicate that the performance consistently improves in the Q+I setting when including the annotations, but diminishes when context is added. Further, we observe that the models demonstrate higher attention to the annotated image compared to the original image (as shown in Fig. 5) and exhibit slightly lower semantic entropy in answering and reasoning, suggesting that the annotations are being processed but not consistently utilized to improve answering accuracy.

To better understand this phenomenon, we examined the reasoning processes of the models to verify whether they observe and comprehend the annotations. Our findings reveal that while the LLaVA model rarely references text annotations during reasoning, the LLaVA-Vicuna and LLaVA-NeXT models frequently incorporate these annotations. For instance, the LLaVA-NeXT model reasons: "The image shows a beaker of blue liquid with a label that says 'Chemical solution.' It is not safe to drink this liquid without knowing its contents and potential hazards. [...]" This example demonstrates that these models are capable of recognizing and interpreting text annotations.

However, even in cases where annotations are acknowledged during reasoning, the models often fail to utilize this information effectively in VQA answering, particularly in the presence of additional context. We observe that while the models can correctly recite annotations when directly queried, they struggle to generalize this information to answer the questions directly.

We interpret this as a limitation in the models' ability to integrate multimodal information consistently, as the context still has large effect and the provided information is not utilized efficiently, only if the model is specifically asked to reason (this observation is, e.g., also made in chain of thought reasoning [1])

[1] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35 (2022): 24824-24837.

2. I don't quite understand why the initial hypothesis is introduced in Section 5.1, as it doesn't seem to be strongly related to the main part of the experiments.

We introduce the first hypothesis to demonstrate the essential role of the image in our setting. The SI-VQA dataset has been designed so that the question cannot in theory be answered without the image modality. Validating the first hypothesis proves that the model does not rely on prior knowledge from pre-training (data leakage) to answer the question, avoiding limitations from previous datasets [1,2]

[1] Yue, Xiang, et al. "Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. 

[2] Chen, Lin, et al. "Are We on the Right Way for Evaluating Large Vision-Language Models?." arXiv preprint arXiv:2403.20330 (2024).

3.  Could the authors explain specifically how GPT-4o is used as an evaluator of reasoning ability? Since the SI-VQA dataset has only 100 samples, why didn't the authors consider using human evaluation instead? Would that provide more accurate results?

Appendix B details how GPT-4o is used as an evaluator of reasoning ability. In reasoning quality evaluation, GPT-4o is prompted to rate the reasoning from 0 to 10, using the prompt: "Rate the explanation's quality from 0 to 10. Give 10 for detailed, well-argued, and correct explanations. Give 0 for a poorly reasoned, wrong, or single-word explanation based on the question and image. Don't rate too harshly, use the full scale and output only the final score". During uncertainty computation, the number of the sampled outputs and the sampling temperature T are set to 10 and 0.9 respectively. We use conditional probabilistic sampling.

As we designed the dataset, we also verified instance by instance the ratings given by GPT4-o for each reasoning. We have observed a very fair alignment with our own judgment and therefore decided to keep GPT4-o evaluation to measure the reasoning quality. We also plotted the distribution of reasoning quality scores by GPT-4o for all LLaVA architectures (see Figure 11 in Appendix H.1). We observed lower values for the standard LLaVA and a high bias towards the quality score of "8" for all models. The positive skewness is certainly due to the data diversity of ratings during GPT4-o training.

Thank you for your comments and for raising these important points. We appreciate the opportunity to clarify our findings and address your concerns.

1. Providing sufficient analysis. [...]

Despite the novelty aspects enumerated in the general answer to all reviewers, the conclusions of our paper are useful for advancing future VQA work:

• Attention is not Explanation. Attention distribution on input tokens does not correlate to VLM performance - see ablation study about using prompt engineering to steer model's attention toward the context and Figure 6. This conclusion from the paper contributes to the ongoing debate about the role of attention and whether it is a form of explanation. Our observations support the

• Attention and Number of Tokens. Our findings suggest that attention to a modality is not necessarily determined by the number of tokens associated with it. For instance, despite LLaVA-Next having approximately six times more image tokens, its attention toward those tokens remains comparable to models with fewer tokens -- like LLaVA-1.5.

A promising avenue for further research is investigating tasks where both modalities provide semantically distinct yet equally essential information. By analyzing scenarios where successful task completion (e.g., answering a question) requires complementary inputs from both modalities, we can deepen our understanding of how models integrate and prioritize multimodal information.

Could you please elaborate more to clarify what findings are not novel, and support that with prior work with similar findings? While our results might adhere to previous work, the uniqueness of the SI-VQA dataset and seven modality configurations offer a completely new context to evaluate modality interplay. While some findings were expected (role of complementary/contradictory context, ...), others are much more surprising and novel (role of annotations, the relation between attention and performance (see ablation study), effect of redundant information, the overconfidence of PaliGemma model,...). 

2.  Size of dataset.

Regarding the dataset size,

• Evaluation purpose. We emphasize that our dataset is only used for evaluation, not for training or fine-tuning models. Its purpose is to analyze pretrained models' behavior when faced with diverse input configurations. 

• TinyBenchmark. While the SI-VQA dataset we designed remains small, prior recent research [1] demonstrates that smaller datasets can still yield meaningful results when the quality of instances is high. Each instance of the SI-VQA Dataset underwent multiple rounds of refinement, ensuring that context descriptions serve only as background information without providing enough detail to answer the question independently.

• Low Variance. Despite the small size, the dataset exhibits low variance in model performance and uncertainty, as demonstrated in the Appendix H. Increasing the number of instances would not alter the conclusions drawn, as they hinge on the modality configurations rather than the dataset size. Instead, rebalancing the role of modalities, such as prioritizing text and introducing spurious visual information, would yield new insights---a direction we are actively exploring.

• Scalability and Community Engagement. To ensure data quality and consistency, we intentionally limited the dataset size to 100 well-curated instances. However, we have also developed a ready-to-use interactive interface that allows the community to generate additional instances with new images and textual descriptions. This ensures that our SI-VQA dataset is extensible and can grow with greater diversity and community input. See Appendix E for details about the Interface and instructions to create new data instances and the link to the code to try it out: https://gitlab.com/dekfsx1/isi-vlm.

[1] tinyBenchmarks: evaluating LLMs with fewer examples

In the revised version of the paper, we integrated this argumentation in the Limitations and Future Directions paragraph in Discussion.

Thank you to the authors for their patient responses, which have resolved some of my doubts. However, considering the overall quality of the paper, I will maintain my original rating.