Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering

The authors have made a series of contributions to the field of Audio-Visual Question Answering (AVQA). Firstly, the introduction of the innovative dataset MUSIC-AVQA-R, along with its associated evaluation metrics, stands out. This dataset and metrics are crucial for a comprehensive evaluation of AVQA models' reasoning capabilities and their ability to generalize across different scenarios. Secondly, their novel AVQA architecture, which incorporates the MCCD strategy to mitigate training biases, is interesting. This marks the first concerted effort to systematically tackle biases in AVQA tasks, examining both model evaluations and designs. Additionally, the authors have undertaken extensive experiments with both MUSIC-AVQA and MUSIC-AVQA-R, providing robust evidence of the proposed architecture and debiasing strategy's effectiveness and superiority. Finally, their evaluation of 13 multimodal QA methods on the new dataset exposes a significant limitation: these methods struggle to generalize effectively, whether within the distribution or in out-of-distribution contexts.

Prevalent Audio-Visual Question Answering (AVQA) methods often learn biases from the dataset, which can lead to reduced robustness. Additionally, current datasets may not accurately assess these methods. To address these issues, the authors introduce MUSIC-AVQA-R, a dataset created in two steps. First, they rephrase questions in the test split of a public dataset (MUSIC-AVQA) and then introduce distribution shifts to the split questions. Rephrasing the questions allows for a diverse test space, while the distribution shifts enable a comprehensive evaluation of robustness for rare, frequent, and overall questions. The authors also propose a robust architecture that uses a multifaceted cycle collaborative debiasing strategy to overcome biased learning.

This paper first analyzes the bias in the existing benchmark (MUSIC-AVQA) on Audio-Visual Question Answering (AVQA). They find MUSIC-AVQA testing set is created using templates and only has 9k QA pairs with limited vocabulary, leading to the potential correlation between words in questions and answers. Therefore, previous methods optimized on MUSIC-AVQA could "memorize" the regularity between questions and answers and may not be proper for AVQA question in real world, which include diverse QA pairs. To evaluate the robustness of the AVQA methods, they use a rephraser to rephrase questions in MUSIC-AVQA testing set, thus create the MUSIC-AVQA-R test set. In addition to the rephrased testing set, they design a model trained with proposed multifaceted cycle collaborative debiasing (MCCD) strategy that decreases the logit between any single modality and multimodality logit. MCCD also includes the loss function to maintain the similarity between bias logits. In experiments, they report results on both MUSIC-AVQA and MUSIC-AVQA-R testing sets. Comparing with previous methods, their proposed approach gets good performances in both MUSIC-AVQA and MUSIC-AVQA-R.

• Reply for dataset quality and bias difference between AVQA and VQA. First, we believe the rephrasing quality of MUSIC-AVQA-R can be guaranteed from three aspects: (1) professional annotators, (2) a strict screening mechanism, and (3) strong support from statistical indicators. For example, as mentioned in Section 3.1 and Appendix B, the Fleiss Kappa value used to measure consistency is 0.839, demonstrating high rephrasing quality. Secondly, we think the suggestion for comparing (train vs. test_head) and (train vs. test_tail) to be very valuable. The results, shown in "response.pdf," provide insightful analysis. In comparing the training and head splits, we observe that the distributions are similar, except for the "comparative" and "existential" questions. However, when comparing the training and tail splits, the distributions differ significantly. Therefore, performance on the head and tail splits can accurately reflect model robustness in both in- and out-of-distribution scenarios. We will also include this comparison result in the revised version of the manuscript. Next, we thank you for highlighting the three debiasing methods for VQA. Specifically, [1] generates various counterfactual training samples by masking critical objects in images or words in questions, assigning different ground-truth answers, thereby forcing VQA models to focus on all critical objects and words. [2] employs a greedy gradient ensemble strategy to eliminate language biases, including distribution and shortcut biases. Similarly, [3] uses a generative adversarial network and knowledge distillation loss to capture both dataset distribution bias and target model bias. To verify the effectiveness of these methods, integrations with non-debiasing VQA methods like UpDn [4] and debiasing methods such as RUBi [5] are conducted. However, we note that these methods are primarily designed for VQA, specifically addressing language bias, rather than for AVQA. The training bias in AVQA contains language, audio, and video bias. Taking STG [6] as an example, the experimental results on the MUSIC-AVQA test set are as follows. We see that this method can achieve 54% overall accuracy only depending on question inputs. It can be seen that relying solely on uni-modal bias can achieve good results. In addition, we can see that only depending on two modalities inputs obtain significant improvement, although AVQA is a task that involves three modalities. These results show that the mentioned training bias plays a key role in predicting answers. Therefore, we believe our method should be compared with both non-debiasing and debiasing AVQA methods. Unfortunately, research on AVQA is still in its infancy, with COCA being the only existing bias elimination method. For this reason, our comparison is limited to non-debiasing AVQA methods. Following COCA [7], we compare our method with audio, visual, video, and AVQA baselines. Due to the lack of public code for COCA, we only compare our method with COCA on the MUSIC-AVQA dataset. Additionally, we think MCCD is specifically designed to mitigate three biases in the AVQA task. This strategy captures the audio, video, and language biases respectively, and then reduces the biases by enlarging the distance between the uni-modal logit and fusion logit, constrained by cycle guidance. To verify MCCD's effectiveness and plug-and-play capability, we conduct extensive experiments on two public datasets. We will discuss the aforementioned three works in the related work section.

• Reply for theoretical analysis of Equation (1). First, we focus on the relationship between distance and bias. In the context of logit magnitudes being constant, an increase in the distance between the uni-modal logit and the fusion logit generally implies a greater difference in their respective probability distributions. For example, in Figure 2 of "response.pdf", the distance between $A$ and $B'$ is larger than the distance between $A$ and $B$, which also makes the distribution difference (i.e. KL divergence) greater. Generally, considering the fusion logit distribution $X$ and a uni-modal logit distribution $Y$, the KL divergence $KL(Y||X)$ tends to increase as the distance between the logits grows. Furthermore, if we consider the mutual information $I(Y;X) = H(X) + H(Y) - H(X,Y)$ between $X$ and $Y$, and utilize the definition of KL divergence, we get $I(Y;X) = H(X) - KL(Y||X)$. This numerically indicates that, while keeping the fusion logit distribution fixed, a distance increase results in a mutual information decrease. This leads to a reduction in correlation between the uni-modal and fusion logit. Several works [8], and [9] have discussed the relationship between mutual information and debiasing, supporting this observation. Next, we will address why we opted to use the reciprocal of distance in MCCD. Our optimization goal is to maximize the separation between the fusion logit and uni-modal logit. Using $\frac{1}{d}$ as a metric allows for heightened sensitivity to small distance variations when the distance is small. Conversely, when a significant distance is established, further variations in distance have a relatively diminished impact. By employing $\frac{1}{d}$, the loss function becomes more effective in encouraging greater separation when logits are close to each other, thus ensuring that small distances are penalized more heavily. Additionally, $-d$ may be unsuitable: (1) may not handle the mentioned scenario, (2) fails to ensure nonnegativity of the total loss, and (3) may reduce the total loss and gradient. We hope this explanation clarifies the rationale behind MCCD and adequately addresses your concerns.

This paper investigates bias in Audio-Visual Question Answering, introducing a new dataset and a method to mitigate the bias. All questions raised were addressed during the post-rebuttal discussion, leading to a consensus in favor of accepting the submission. The AC finds no significant concerns and agrees with this assessment.