Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality

The choice of base models is not comprehensive. The authors selected LLaVA-1.5 and Qwen2-VL as base models to validate their method, but introducing more models would help confirm the method’s universality. For instance, a table could be added to the appendix to show the method’s cross-model generalizability. (part.2)

To demonstrate the effectiveness of our approach on large multimodal language models of different architectures, we added experimental data from the Q-former-based InstructBLIP model and the embedding-autoregressive-based Chameleon model to the original experimental data from the vision encoder-mlp-llm paradigm. On the Chameleon model, our method comprehensively surpasses the performance of the baseline model and effectively suppresses the multimodal hallucinations of the model.

InstructBLIP:

We are deeply grateful for the your recognition of our work's innovation and thoroughness, as well as their constructive feedback. We have addressed each suggestion on the manuscript's weaknesses and made the necessary revisions.

How was the hyperparameter (e.g., $\gamma$) determined? Were hyperparameters independently tuned for different benchmarks and model/method variants?

$\gamma$ represents the degree of confidence in the treatment effect and is used to adjust the strength of suppressing the modal prior. We have added this explanation in the text. For different models, the hyperparameter settings that achieve optimal performance are similar.

The choice of base models is not comprehensive. The authors selected LLaVA-1.5 and Qwen2-VL as base models to validate their method, but introducing more models would help confirm the method’s universality. For instance, a table could be added to the appendix to show the method’s cross-model generalizability.

To demonstrate the effectiveness of our approach on large multimodal language models of different architectures, we added experimental data from the Q-former-based InstructBLIP model and the embedding-autoregressive-based Chameleon model to the original experimental data from the vision encoder-mlp-llm paradigm. On the Chameleon model, our method comprehensively surpasses the performance of the baseline model and effectively suppresses the multimodal hallucinations of the model.

Chameleon:

Section 3.1 and Figure 2 are hard to follow. Section 3.1 introduces very complex causal relationships, but these seem not closely tied to the method. If I understand correctly, the proposed method is largely similar to VCD: it generates counterfactual logits by perturbation and improves predictions by removing this "background noise" from the normal logits, which does not necessitate such complex preliminary explanations. I would also like to mention that the discussed in Section 3.3 might be unnecessary.

We believe that detailed causal modeling can greatly help analyze the connections and independent effects of different factors in the model.

Specifically, VCD contrasts outputs derived from original and distorted image inputs. In the contrast, CausalMM isolates the influence of modal priors and other confounders on multimodal attention by using backdoor adjustment methods, obtains the positive treatment effect of attention on output through counterfactual reasoning, adjusts the output of the model at the attention and feature levels, and balances the modality priors. The two are fundamentally different at all levels.

Tabular Comparison of CausalMM and VCD

We believe that the discussion of ${P_{effect}}$ in section 3.3 is necessary because they represent the paradigm of counterfactual reasoning in causal theory and fully present the complete process of the do operator and counterfactual reasoning. This is necessary to help readers understand how we obtain the positive causal effect of effective attention on the model output.

The equations are also hard to follow. For example, in Section 3.3, the core of the method involves logits processing, which is unrelated to softmax. Therefore, separating logits from the softmax equation and presenting them independently would be clearer and more concise.

Thank you for your suggestion. We think the argmax function can more intuitively show our selection process for the next token in the output sequence. If you have more suggestions on the writing format, please let us know.

We greatly appreciate your thoughtful critique and suggestions. Below is a summary of our revisions and clarifications based on your feedback:

• Cross-model generalizability: As per your request, we provided experimental data in "Author Response to Reviewer gJXZ (Part 1)" and "Author Response to Reviewer gJXZ (Part 2)" demonstrating the performance of our method on Meta's Chameleon model as well as the InstructBLIP model. We have also included experimental results for InstructBLIP and Chameleon in the appendix, which confirm that our method is applicable across several mainstream MLLM architectures. We encourage you to review these additions.

• Applicability and generalizability: Following your suggestions, we discussed hyperparameter-related details in "Author Response to Reviewer gJXZ (Part 1)" and added explanations regarding hyperparameters in line 234 of the revised paper. Additionally, we conducted hyperparameter sensitivity tests for our method, with the detailed data provided in "Author Response to Reviewer CnJZ (Part 3)".

• Differences from the VCD method: In response to your feedback, we elaborated on the differences between our method and the VCD method in "Author Response to Reviewer gJXZ (Part 3)". Furthermore, we added corresponding content in the appendix of the paper to detail the theoretical derivations related to causal reasoning and reiterated the importance of structural causal modeling.

We hope these revisions and clarifications address your concerns and look forward to any additional feedback or questions.

Dear Reviewer gJXZ,

Thank you for your valuable time! We are writing to kindly follow up on the status of our manuscript review. We have been actively engaging with other reviewers and have received valuable feedback. Your insights would be greatly appreciated to further enhance the quality of our work.

Thank you for your time and consideration!

Yours sincerely,

CausalMM Team

Dear Reviewer gJXZ,

Thank you for your valuable feedback and for taking the time to review our manuscript!

Best regards,

CausalMM Team

We are deeply grateful for your recognition of our work's innovation and thoroughness, as well as their constructive feedback. We have addressed each suggestion on the manuscript's weaknesses and made the necessary revisions.

Insufficient Theoretical Justification: The paper lacks a deep theoretical analysis of why the proposed causal interventions lead to improved performance. The causal model is described, but the theoretical foundations and assumptions are not thoroughly explored or justified.

We have supplemented some arguments related to the causal inference theory in the paper. We will describe how our causal inference method balances the model's modal priors from three aspects: structural causal model, backdoor adjustment, and counterfactual reasoning:

Structural Causal Model (SCM)

The process of methods based on causal theory includes building the structural causal models and choosing the proper way of causal inference. In the structure, nodes indicate the important variables or variables that are hidden but have an effect on the important ones, while the edges indicate inner causal relationships. Combining different causal structures and different concerns, suitable causal reasoning methods are selected to eliminate spurious correlations.

Backdoor Adjustment

In this work, we reinterpret the backdoor adjustment framework to analyze the causal influence of modality priors on attention mechanisms and model outputs. By identifying modality priors ($M$) as confounders, we isolate the causal effect of attention ($A$) on the output ($O$) using the backdoor adjustment method.

Variables:

$A$ (attention): The mechanism whose causal effect we aim to evaluate.

$M$ (modality priors): Influences both $A$ and $O$, acting as a confounder.

$O$ (model output): The outcome variable influenced by $A$ and $M$.

Causal Challenge:

The backdoor path $A \leftarrow M \to O$ introduces confounding, making it necessary to adjust for $M$ to isolate the causal effect of $A$ on $O$.

To block confounding, the backdoor criterion ensures that:

1. $M$ blocks all backdoor paths from $A$ to $O$.

2. $M$ is not influenced by $A$.

Using this criterion, the causal effect of $A$ on $O$ is computed as

$P(o \mid do(a)) = \sum_m P(o \mid a, m) P(m),$

Modality priors ($M$) explain the indirect influence of $A$ on $O$, enabling a disentangled analysis. Adjusting for $M$ removes the confounding, ensuring that $A$'s causal impact on $O$ is properly estimated.

Counterfactual reasoning

By controlling for confounders, we can more accurately estimate causal relationships. This method serves as a foundation for counterfactual reasoning, which enables the assessment of treatment effects in systems like multimodal models.

Causal Effect of Visual Attention ($A_i$)

The causal effect of the visual attention mechanism on the model output $O$ is given by:

$P_{effect\_V} = E_{A_i \sim \tilde{A}_i}\left[P(O | A_i = **A**_i, I = **I**, P_v = **P**_v) - P(O | \text{do}(A_i = **a**_i), I = **I**, P_v = **P**_v)\right].$

Here:

$P_{effect\_V}$ represents the treatment effect of visual attention on the output $O$.

$**A**_i$ denotes the observed visual attention, while $**a**_i$ represents the intervention applied to the visual attention.

Causal Effect of Language Model Attention ($A_t$)

Similarly, the causal effect of the language model attention on the output $O$ can be expressed as:

$P_{effect\_L} = E_{A_t \sim \tilde{A}_t}\left[P(O | A_t = **A**_t, T_t = **T**_t, P_l = **P**_l) - P(O | \text{do}(A_t = **a**_t), T_t = **T**_t, P_l = **P**_l)\right],$

Here:

$P_{effect\_L}$ represents the treatment effect of language attention on the output $O$.

$**A**_t$ is the observed language model attention, while $**a**_t$ represents the intervention on the language model attention.

Combined Causal Effect in a Multimodal Setting

In multimodal systems, the combined treatment effect of both visual and language attention mechanisms is described as:

$P_{effect\_M} = E_{A_i, A_t \sim \tilde{A}_i, \tilde{A}_t}\left[P(O | A_i = **A**_i, A_t = **A**_t, I = **I**, T_t = **T**_t, P_v = **P**_v, P_l = **P**_l)\right] - P(O | \text{do}(A_i = **a**_i), \text{do}(A_t = **a**_t), I = **I**, T_t = **T**_t, P_v = **P**_v, P_l = **P**_l),$

In this formulation:

$P_{effect\_M}$ measures the combined effect of both visual and language attention mechanisms on the model output.

The observed and intervened attention variables are denoted by $**A**_i, **a**_i$ for visual attention, and $**A**_t, **a**_t$ for language attention.

Several claims lack explanations and references. The authors are advised to carefully proofread the paper and add the necessary citations. For instance: Line 65: “Modality priors are one of the confounding factors in the causal path of MLLM.” Why? And what is the definition of the confounding factor? Line 58-61: “existing decoding strategies……, overlooking the causal relationships among attention visual attention, language attention, modality priors, and model output.

We added references to relevant articles to support our point of view. At the same time, for the possible confusion, we modified the sentence in the original text to make the conclusion more moderate.

Confounding Factor: A confounding factor, also known as a confounder, is a variable that influences both the dependent variable and independent variable, causing a spurious association. In simpler terms, it's an outside influence that can distort the true relationship between the variables being studied.

Without causal inference theory, the true impact of factors such as attention on model output under the influence of modal priors has not been well studied.

[1] Peng, D., Wei, W., Mao, X., Fu, Y., Chen, D. "An Empirical Study on the Language Modal in Visual Question Answering". arXiv preprint arXiv:2305.10143

[2] Chen, M., Cao, Y., Zhang, Y., Lu, C. "Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective". arXiv preprint arXiv:2403.18346

[3] Lukics, K. S., Lukács, Á. "Modality, presentation, domain and training effects in statistical learning". Sci Rep, 2022.

[4] Gema, A. P., Jin, C., Abdulaal, A., Diethe, T., Teare, P., Alex, B., Minervini, P., Saseendran, A. "DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations". arXiv preprint arXiv:2410.18860

[5] Lee K, Kim M, Yoon S, et al. VLind-Bench: Measuring Language Priors in Large Vision-Language Models. arXiv preprint arXiv:2406.08702, 2024.

In Section 3.3, the authors introduce the causal effect and the corresponding calculation of selecting the next token. But what’s next after determining the index of the next token? The authors are advised to supplement more on this point. I assume that it is going to talk about inferencing with the modified next token prediction.

We have added content about the selection of the next token in the revised version. We use direct sampling as a decoding strategy because it can improve the diversity of text and reduce the probability of repeatedly outputting the same word.

Lack of implementation details. It would be more convincing to provide more details on how to reproduce the experimental results.

We are already preparing an open source version of the code to increase readability. Please pay attention to our follow-up work.

The notations should be considered more carefully. For example, in Section 3.3, it is confusing that in Line 222-223 represents the original output but in Line 226-227 it denotes the model output.

Thank you for your feedback. We have corrected the representation of the relevant variables in the revised version.

What does $\gamma$ mean in the equations in Section 3.3?

$\gamma$ represents the degree of confidence in the treatment effect and is used to adjust the strength of suppressing the modal prior. We have added this explanation in the text.

Could you provide the results of VCD and OPERA on VLind-Bench?

We provide experimental data of VCD and OPERA methods on the VLind benchmark. The result of the OPERA method is 0, which is similar to some models in Table 1 in the original VLind paper[1]. No conclusion is given in the original paper. The reason is speculated to be the decline in the ability to follow certain instructions.

[1] Lee K, Kim M, Yoon S, et al. VLind-Bench: Measuring Language Priors in Large Vision-Language Models[J]. arXiv preprint arXiv:2406.08702, 2024.

In Lines 58-61, the authors state that existing decoding strategies overlook the causal relationships among attention visual attention, language attention, modality priors, and model output. However, VCD [1] injects noise into the image input and acquires the output logits for contrastive decoding, which can be seen as a counterfactual intervention at the visual input. The effects of this intervention are also later shown in the attention parts and model outputs [2,3]. Could the authors make more discussion on this perspective?

Thank you for your feedback. Specifically, VCD contrasts outputs derived from original and distorted image inputs. In the contrast, CausalMM isolates the influence of modal priors and other confounders on multimodal attention by using backdoor adjustment methods, obtains the positive treatment effect of attention on output through counterfactual reasoning, adjusts the output of the model at the attention and feature levels, and balances the modality priors. The former's operation is not consistent with the definition of intervention in the causal theory system. Causal reasoning does not directly use intervention to optimize the output.

Tabular Comparison of CausalMM and VCD

We are deeply grateful for your recognition of our work's innovation and thoroughness, as well as their constructive feedback. We have addressed each suggestion on the manuscript's weaknesses and made the necessary revisions.

Insufficient Theoretical Justification: The paper lacks a deep theoretical analysis of why the proposed causal interventions lead to improved performance. The causal model is described, but the theoretical foundations and assumptions are not thoroughly explored or justified.

We have supplemented some arguments related to the causal inference theory in the paper. We will describe how our causal inference method balances the model's modal priors from three aspects: structural causal model, backdoor adjustment, and counterfactual reasoning:

Structural Causal Model (SCM)

The process of methods based on causal theory includes building the structural causal models and choosing the proper way of causal inference. In the structure, nodes indicate the important variables or variables that are hidden but have an effect on the important ones, while the edges indicate inner causal relationships. Combining different causal structures and different concerns, suitable causal reasoning methods are selected to eliminate spurious correlations.

Backdoor Adjustment

In this work, we reinterpret the backdoor adjustment framework to analyze the causal influence of modality priors on attention mechanisms and model outputs. By identifying modality priors ($M$) as confounders, we isolate the causal effect of attention ($A$) on the output ($O$) using the backdoor adjustment method.

Variables:

$A$ (attention): The mechanism whose causal effect we aim to evaluate.

$M$ (modality priors): Influences both $A$ and $O$, acting as a confounder.

$O$ (model output): The outcome variable influenced by $A$ and $M$.

Causal Challenge:

The backdoor path $A \leftarrow M \to O$ introduces confounding, making it necessary to adjust for $M$ to isolate the causal effect of $A$ on $O$.

To block confounding, the backdoor criterion ensures that:

1. $M$ blocks all backdoor paths from $A$ to $O$.

2. $M$ is not influenced by $A$.

Using this criterion, the causal effect of $A$ on $O$ is computed as

$P(o \mid do(a)) = \sum_m P(o \mid a, m) P(m),$

Modality priors ($M$) explain the indirect influence of $A$ on $O$, enabling a disentangled analysis. Adjusting for $M$ removes the confounding, ensuring that $A$'s causal impact on $O$ is properly estimated.

Counterfactual reasoning

By controlling for confounders, we can more accurately estimate causal relationships. This method serves as a foundation for counterfactual reasoning, which enables the assessment of treatment effects in systems like multimodal models.

Causal Effect of Visual Attention ($A_i$)

The causal effect of the visual attention mechanism on the model output $O$ is given by:

$P_{effect\_V} = E_{A_i \sim \tilde{A}_i}\left[P(O | A_i = **A**_i, I = **I**, P_v = **P**_v) - P(O | \text{do}(A_i = **a**_i), I = **I**, P_v = **P**_v)\right].$

Here:

$P_{effect\_V}$ represents the treatment effect of visual attention on the output $O$.

$**A**_i$ denotes the observed visual attention, while $**a**_i$ represents the intervention applied to the visual attention.

Causal Effect of Language Model Attention ($A_t$)

Similarly, the causal effect of the language model attention on the output $O$ can be expressed as:

$P_{effect\_L} = E_{A_t \sim \tilde{A}_t}\left[P(O | A_t = **A**_t, T_t = **T**_t, P_l = **P**_l) - P(O | \text{do}(A_t = **a**_t), T_t = **T**_t, P_l = **P**_l)\right],$

Here:

$P_{effect\_L}$ represents the treatment effect of language attention on the output $O$.

$**A**_t$ is the observed language model attention, while $**a**_t$ represents the intervention on the language model attention.

Combined Causal Effect in a Multimodal Setting

In multimodal systems, the combined treatment effect of both visual and language attention mechanisms is described as:

$P_{effect\_M} = E_{A_i, A_t \sim \tilde{A}_i, \tilde{A}_t}\left[P(O | A_i = **A**_i, A_t = **A**_t, I = **I**, T_t = **T**_t, P_v = **P**_v, P_l = **P**_l)\right] - P(O | \text{do}(A_i = **a**_i), \text{do}(A_t = **a**_t), I = **I**, T_t = **T**_t, P_v = **P**_v, P_l = **P**_l),$

In this formulation:

$P_{effect\_M}$ measures the combined effect of both visual and language attention mechanisms on the model output.

The observed and intervened attention variables are denoted by $**A**_i, **a**_i$ for visual attention, and $**A**_t, **a**_t$ for language attention.

Limited Novelty: While the application of causal inference to MLLMs is presented as novel, causal reasoning has been previously applied in machine learning models, including language models. The paper does not sufficiently differentiate its contributions from existing work in causal inference applied to deep learning.

While causal inference has been applied in machine learning, the novelty of CausalMM lies in its specifically designed structural causal model and counterfactual reasoning framework tailored for Multimodal Large Language Models (MLLMs). Unlike previous works, CausalMM explicitly addresses the causal relationships between visual and language modalities and identifies and adjusts the influence of modality priors on attention mechanisms through intervention.

[1] Gong, W., Jennings, J., Zhang, C., Pawlowski, N. "Rhino: Deep Causal Temporal Relationship Learning With History-dependent Noise". Machine Learning, 2022.

[2] Fang, Y., Liang, F. "Causal-StoNet: Causal Inference for High-Dimensional Complex Data". arXiv preprint arXiv:2403.18994

[3] Cheng, Y., He, K., Xiao, T., Dai, Q., Suo, J., Li, Z., Yang, R. "CUTS: Neural Causal Discovery from Irregular Time-Series Data". International Conference on Learning Representations, 2023.

[4] Voloch, L., Barton, R. A., Ioannidis, V., Donno, C. D., Wu, Y., Price, L., Karypis, G., Wang, Z. "Predicting Cellular Responses with Variational Causal Inference and Refined Relational Information". International Conference on Learning Representations, 2022.

Inadequate Comparison with Baselines: For some evaluations(such as figure 3,5,6), the experimental evaluation compares the proposed method with only a partial set of baselines from the setup. Under many settings, the performance of this method underperform or only slightly outperform baseline methods as shown in table 1.

We added this table to expand the comparison with more baselines. In the table, the values taken are the averages of the three parts of the POPE benchmark (MSCOCO, A-OKVQA, GQA). It can be seen that the CausalMM method can achieve the highest value most of the time.

Superficial Experimental Analysis: The results, while showing improvements, lack statistical significance testing. Additionally, there is a lack of detailed analysis of where and why the method improves performance, making it difficult to assess the true impact.

We performed a statistical significance analysis on the experimental results. The significance level is 0.05. The results of the statistical significance test support the advantages of our method and prove its effectiveness in practical applications.

Where and why the method improves performance:

Multimodal large language models (MLLMs) are prone to hallucination problems caused by modality priors (visual or language), such as relying on textual cues and ignoring visual input. The model may mistakenly rely on the attention distribution of a specific modality, leading to incorrect cognition of object existence and attributes.

CAUSALMM analyzes the causal influence of visual and language attention through structured causal modeling (SCM), considers modality priors as confounding factors, and corrects causal paths through counterfactual reasoning. By generating different attention states (such as random, reversed, and unified attention) through counterfactual interventions, the contributions of different modalities can be isolated and quantified, thereby more accurately judging the causal role of multimodal information.

In the multimodal collaborative mode, CAUSALMM balances the causal effects of visual and language attention to make the generated content consistent with the multimodal input, thereby improving the generation quality of the model.

Lack of Discussion on Limitations: The paper does not adequately discuss the limitations or potential downsides of the proposed method, such as scenarios where it might not work well or possible negative impacts. & Limitations and Failure Cases: What are the limitations of your method? Are there scenarios where it does not perform well or might even degrade performance? How do you address potential negative impacts?

We added more discussions on limitations in the appendix. In the case study section of Section 4.4 of the original paper, we show examples that our method still fails to solve. These examples focus on fine-grained visual perception, perception of spatiotemporal relations, and understanding of high-order semantics. As a balancer of modal priors, CausalMM can maximize the capabilities of existing backbone models, but its limitations lie in the performance limits of the vision encoder and LLM. We will continue to study how to effectively utilize the existing visual information and maximize the performance of the vision encoder and LLM.

Computational Complexity: What is the computational overhead introduced by your method? How does it compare to the baseline models in terms of runtime and resource consumption?

Similar to other methods that optimize at the inference stage, our method will consume more time in the complete inference process. The additional time is negligible in the latency of normal conversations.

Impact of Hyperparameters: How sensitive is your method to the choice of hyperparameters involved in the interventions? Have you performed a sensitivity analysis?

Our method is not sensitive to non-marginal values of hyperparameters. Below is our sensitivity analysis of some experimental results.

Sensitivity Table (Gamma, Epsilon)

Robustness to Noise: How does your method handle noisy or adversarial inputs? Does the causal framework improve robustness in such cases?

CausalMM can handle noisy and adversarial inputs. Noise can be included in counterfactual attention together with low-quality attention and desensitized through the framework of CausalMM. For adversarial inputs, the settings of VLind benchmark and POPE benchmark include such inputs. From the experimental results of the original paper, CausalMM can handle adversarial inputs well and improve the alignment of model output with input.

Applicability and Generalization: The approach is tested on specific MLLMs, but it is unclear how well the method generalizes to other models. & Generality of the Approach: How well does your method generalize to other types of MLLMs, such as chameleon?

To demonstrate the effectiveness of our approach on large multimodal language models of different architectures, we added experimental data from the Q-former-based InstructBLIP model and the embedding-autoregressive-based Chameleon model to the original experimental data from the vision encoder-mlp-llm paradigm.

Chameleon:

In addition, the experimental results of InstructBLIP can be found in the Appendix.

We greatly appreciate your thoughtful critique and suggestions. Below is a summary of our revisions and clarifications based on your feedback:

• Theoretical support for the validity of causal reasoning: In "Author Response to Reviewer CnJZ (Part 1)", we provided a comprehensive justification for the validity of causal reasoning. Correspondingly, we have improved potentially confusing sections in the revised version of the main text and added relevant theoretical derivations and justifications in the appendix.

• Evidence for innovation and effectiveness: In "Author Response to Reviewer CnJZ (Part 3)", we presented comparisons between our method and several others. Additionally, in "Author Response to Reviewer CnJZ (Part 4)", we provided experimental data demonstrating the performance of our method on Meta's Chameleon model. We have also added experimental results for InstructBLIP and Chameleon in the appendix, which show that our method is applicable across several mainstream MLLM architectures. We welcome you to check these additions.

• Applicability and generalizability: At your request, we conducted hyperparameter sensitivity tests and statistical significance analyses of our method. The specific data has been detailed in "Author Response to Reviewer CnJZ (Part 3)".

• Discussion of limitations: In accordance with your request, we have added a discussion of the limitations of our method, along with corresponding content in the appendix of the paper.

We hope these revisions and clarifications address your concerns and look forward to any additional feedback or questions.

Dear Reviewer CnJZ,

Thank you for your positive response and support for our work! We noticed that the rating has not been updated, so we would like to confirm this with you.

Thank you again for your time and assistance!

Yours sincerely,

CausalMM Team

Thank you for providing these results. After futher reviewing them in detail, I noticed the following:

1. For the LLaVA-1.5 section, the table appears to simply aggregate results from Table 1, which doesn’t introduce any new insights. This could be misleading, as the three datasets differ in size. Based on the main results from Table 1 in the paper, OPERA still demonstrates a best F1 score on both MSCOCO and GQA.

2. Could you clarify the origin of DOLA? I read through the paper and the discussions, but I couldn’t find any explanation or citations for it.

Additionally, I have a few questions regarding specific figures:

• What are the results of VCD and OPERA on VLind-Bench for the LLaVA and Qwen2-VL models in Figure 3?
• What are the results of OPERA in Figure 4?
• What are the results of OPERA in Figure 5?

I’d appreciate your clarification on these points. Thank you!

Please double check all results, these mistakes in results presented can be misleading.

Thank you for your reminder! We fully understand the importance of data accuracy. We have thoroughly checked for any potential errors caused by LaTeX formatting and ensured the correctness of all results.

Thank you for response. While you provided results for DOLA, could you clarify why no results were included for OPERA?

Of course, we are happy to answer your questions.

LLaVA-1.5 / VLind-Bench

Qwen2-VL / VLind-bench

These two tables should be the ones you have questions about. The results for the OPERA method are not missing, but rather all are zero. This phenomenon also appears for other models in Table 2 on page 7 of the VLind-Bench paper [1]. The original paper does not provide an explanation or conclusion for this, but we speculate that certain methods may have reduced the model's ability to follow specific instructions, leading to responses that result in a final score of zero in the evaluation pipeline. In the next work, we will explore the internal logic related to this phenomenon and deeply reveal the decisive factors related to the ability of the model to follow instructions. Welcome to pay attention to our work in the future!

[1] Lee K, Kim M, Yoon S, et al. VLind-Bench: Measuring Language Priors in Large Vision-Language Models[J]. arXiv preprint arXiv:2406.08702, 2024.

We hope the above explanation has resolved your doubts. Thank you for your feedback and support!

CausalMM Team

Please explain which paths need to be blocked in Figure 2. It appears that the path from the image to visual attention and from text token embedding to LLM attention should be blocked, but aren’t those paths essential?

Specifically, in addition to the paths from image to visual attention and from text token embedding to LLM attention, the paths from visual modality prior to visual attention and from language modality prior to LLM should also be blocked, but the paths are too short to be well presented as dashed lines in the figure. We have modified the figure to ensure readability.

Regarding the question of whether these paths are essential, this involves our definition of the blocking operation in the backdoor adjustment in the previous question. For example, blocking the path from image I to attention A can be understood as artificially giving a value to A so that the value of A is no longer affected by I. This operation is called the do operation in the causal inference theory system: $do(a)$. The truncation of this path can be expressed as: $P(I \mid do(A)) = P(I)$. The same applies to other cases.

This method seems closer to a contrastive decoding approach rather than backdoor adjustment. It appears to use counterfactual attention similarly to how negative samples are used.

Thank you for your feedback. We have explained back-door adjustment in detail in the first question. Below is a table that distinguishes our method from Contrastive decoding. Specifically, Contrastive decoding contrasts outputs derived from original and distorted inputs. In the contrast, CausalMM isolates the influence of modal priors and other confounders on multimodal attention by using backdoor adjustment methods, obtains the positive treatment effect of attention on output through counterfactual reasoning, adjusts the output of the model at the attention and feature levels, and balances the modality priors.

We use VCD [1] for specific comparison:

Tabular Comparison of CausalMM and VCD

[1] Leng, Sicong, et al. "Mitigating object hallucinations in large vision-language models through visual contrastive decoding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

The intervention on attention to reduce the influence of defined priors as confounders seems to also reduce the impact of input on attention.

Your understanding is correct. But the influence obtained after intervention is not directly positively correlated with the result. The attention and corresponding features obtained through intervention and backdoor adjustment serve as causal probability anchors, helping us dynamically obtain the positive processing effect of effective attention on model output through counterfactual reasoning. It is used to balance modal priors at the feature and hidden states levels. The corresponding formula in the article is as follows:

For the visual attention ($A_i$):

$P_{effect\_V} = E_{A_i \sim \tilde{A}_i}\left[P(O | A_i = **A**_i, I = **I**, P_v = **P**_v) - P(O | \text{do}(A_i = **a**_i), I = **I**, P_v = **P**_v)\right].$

Here, $P_{effect\_V}$ represents the causal effect of the visual attention mechanism on the model output $O$. The term $**A**_i$ denotes the observed visual attention, whereas $**a**_i$ represents the intervention applied to the visual attention.

For the LLM attention ($A_t$):

$P_{effect\_L} = E_{A_t \sim \tilde{A}_t}\left[P(O | A_t = **A**_t, T_t = **T**_t, P_l = **P**_l) - P(O | \text{do}(A_t = **a**_t), T_t = **T**_t, P_l = **P**_l)\right],$

where $P_{effect\_L}$ denotes the causal effect of the language model attention on the output $O$. The notation $**A**_t$ is the observed language model attention, and $**a**_t$ is the intervention applied to the language model attention.

In a multimodal setting, the combined causal effect is given by:

$P_{effect\_M} = E_{A_i, A_t \sim \tilde{A}_i, \tilde{A}_t}\left[P(O | A_i = **A**_i, A_t = **A**_t, I = **I**, T_t = **T**_t, P_v = **P**_v, P_l = **P**_l)\right] - P(O | \text{do}(A_i = **a**_i), \text{do}(A_t = **a**_t), I = **I**, T_t = **T**_t, P_v = **P**_v, P_l = **P**_l),$

where $P_{effect\_M}$ represents the combined causal effect of both visual and language attention mechanisms on the output $O$.

Line 58 mentions the VCD method as considering only statistical correlation. I thought the VCD method also accounts for causation by using negative images to identify language priors. Why do you view the VCD method as overlooking causal relationships?

We think that our statement may have caused confusion, so we adjusted it in the paper. If we talk about causality in a broad sense, your idea should be correct. But based only on the narrow causal reasoning theory system, VCD does not align with any causal model.

On page 5, does 'j' refer to the order of the method for creating counterfactual attention?

In the CausalMM method, "j" iterates over all tokens in the denominator (to compute the softmax normalization). We have added relevant instructions in the revision.

We are deeply grateful for the your recognition of our work's innovation and thoroughness, as well as their constructive feedback. We have addressed each suggestion on the manuscript's weaknesses and made the necessary revisions.

It is unclear if this method constitutes a backdoor adjustment. My understanding of backdoor adjustment involves identifying a confounder, then using it to reduce the confounder's impact, which differs from using counterfactuals as described here. Counterfactuals are typically used to measure natural direct effects or natural indirect effects, and I am curious if this method follows such an approach. Mathematical proof may be needed for line 225.

Thank you for your interest in the relevant theory. In our work, backdoor adjustment and counterfactual reasoning are combined to drive the entire mechanism. The role of backdoor adjustment is to identify confounding factors such as modal priors and reduce the impact of confounding factors on identifying the causal relationship between attention and model output. The state of the variables can be qualitatively described as "counterfactuals", which depends on the specific circumstances of different studies. The counterfactuals you describe should be the counterfactual reasoning we use later, which has been widely used in the field of machine learning. We use counterfactual reasoning to obtain the positive treatment effect brought by intervening attention. The relevant formulas have been described in detail in section 3.3.

The following is a further explanation of backdoor adjustment in our work:

Structural Causal Model (SCM)

Variables and their roles:

$A$ (attention): This represents the model's attention mechanism that we aim to evaluate or manipulate.

$M$ (modality priors): Modality priors influence both the model's attention ($A$) and the output ($O$), thus creating confounding.

$O$ (model output): The outcome variable, which is affected both directly by $A$ and indirectly through $M$.

Causal structure and backdoor paths:

The backdoor path in this SCM is $A \leftarrow M \to O$, which starts with an arrow pointing into $A$ and creates a confounding junction structure.

To isolate the causal effect of $A$ on $O$, the confounding influence of $M$ must be blocked.

Backdoor Criterion:

To apply backdoor adjustment, the adjustment set $M$ must satisfy the following criteria:

1. $M$ blocks all backdoor paths from $A$ to $O$.

2. $M$ does not include any descendants of $A$ (i.e., variables causally influenced by $A$).

By intervening on $A$ and adjusting for $M$, we can isolate the causal effect of $A$ on $O$.

Backdoor Adjustment Formula:

Given a sufficient adjustment set $M$, the causal effect $P(o \mid do(a))$ is identified as:

$P(o \mid do(a)) = \sum_m P(o \mid a, m) P(m)$

Derivation:

1. Starting with the interventional distribution:

$P(o \mid do(a)) = \sum_m P(o \mid do(a), m) P(m \mid do(a))$

2. Using the property of the intervention $do(a)$:

Under the intervention $do(a)$, the variable $A$ is no longer influenced by $M$. Thus:

$P(m \mid do(a)) = P(m)$

3. Replacing $P(o \mid do(a), m)$ with the observational counterpart:

Due to the backdoor criterion, $M$ blocks all confounding paths, allowing:

$P(o \mid do(a), m) = P(o \mid a, m)$

4. Combining these results: $P(o \mid do(a)) = \sum_m P(o \mid a, m) P(m)$

Application to Attention-Output Framework:

In the context of our framework:

1. Backdoor path:

The backdoor path $A \leftarrow M \to O$ reflects the confounding effect of modality priors ($M$) on the attention mechanism ($A$) and the model's output ($O$).

2. Intervention:

By intervening on $A$, we ensure that the causal effect of attention on the output is isolated, free from the influence of modality priors.

3. Adjustment:

To block the backdoor path, we adjust for $M$, computing the summation over all possible values of $M$ to account for its confounding effect.

Full Formula for the Framework:

In our framework, the causal effect of attention ($A$) on the model output ($O$) can be computed as:

$P(o \mid do(a)) = \sum_m P(o \mid a, m) P(m)$

$P(o \mid a, m)$: The conditional probability of the output given attention $A$ and modality priors $M$.

$P(m)$: The marginal probability of modality priors $M$.

We added the corresponding content in the appendix of the revision.

[1] Judea Pearl. Causality. Cambridge university press, 2009.

[2] Kexuan Zhang, Qiyu Sun, Chaoqiang Zhao, and Yang Tang. Causal reasoning in typical computer vision tasks. arXiv:2307.13992, 2023a.

We are delighted to clarify the differences between our method and VCD. Thank you for recognizing our rebuttal.

We greatly appreciate your willingness to discuss the theories related to causality. We fully agree with your interpretation of backdoor adjustment and value your understanding of the related theory. However, some important misunderstandings appear to remain unresolved. The following points should address these concerns:

• In our method, backdoor adjustment and counterfactual reasoning are two closely related but different stages (as we have explained in our responses to Reviewers CnJZ and vNn2). These two theories are applied in different processes and at different scales.

• You may have misunderstood that our backdoor adjustment and counterfactual reasoning describe the same process. For example, the part you mentioned in line 224 pertains to counterfactual reasoning rather than backdoor adjustment. The formula for backdoor adjustment, as mentioned earlier (using example variables), is:

$P(o \mid do(a)) = \sum_m P(o \mid a, m) P(m)$

• In the initial version of our paper, we did not provide a detailed explanation of backdoor adjustment due to space limitations. This might have led to some confusion. Therefore, in the revised version, we added a detailed explanation of backdoor adjustment in the appendix and clarified it in the main text to prevent further misunderstanding. These additions are highlighted in the revised manuscript for your reference.

• Specific distinctions between the two stages in our method:

  • Backdoor adjustment supports the theoretical justification for our method’s ability to estimate the causal effects of other variables despite the confounding factor of modality priors. The key variable in this stage is the modality prior.

  • Counterfactual reasoning involves estimating the causal effect of attention on the model's output under the assumption that attention fails (i.e., using counterfactual attention). The key variable in this stage is attention.

• You interpreted the theorem presented in line 224 as representing direct effect, but it actually represents counterfactual reasoning (as we have explained to other reviewers). The key distinction is that counterfactual reasoning estimates the effect of an event under the hypothetical condition that it does not occur (fails), while direct effect refers to the isolated effect of a variable on the outcome when other variables are held constant [1]. Our method estimates the causal effect of attention on the model’s output under the assumption of attention failure. If we were to use direct effect, we would instead focus on the influence of other variables on the outcome when attention is controlled.

• Regarding your statement that VCD is similar to direct effect, we agree. However, VCD does not constitute counterfactual reasoning. Specifically:

  • In counterfactual reasoning, interventions isolate the influence of specific variables by controlling the values of certain nodes in the causal graph. In contrast, VCD utilizes distorted inputs merely to amplify input biases, rather than to simulate hypothetical conditions.

  • Counterfactual reasoning aims to measure the causal effect of a specific variable, while VCD’s process leans more toward statistical adjustment rather than causal modeling.

[1] Judea Pearl. Causality. Cambridge University Press, 2009.

We hope these clarifications help resolve any misunderstandings about our work. We are also delighted to engage in discussions on causal reasoning, as we believe it is a theoretical framework deserving more attention.

Should you have further suggestions for improving the paper, please let us know! If we have addressed your concerns, we hope you would reconsider the rating.

CausalMM Team