AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models

We thank reviewer huna for recognizing our work as providing a new perspective on multimodal hallucination, and for delivering a well-written and clear paper, along with constructive feedback. We address the concerns raised by reviewer huna below, and encourage the reviewer huna to check the newly updated revision PDF that includes the requested content. The red-colored texts indicate the newly added part in the revision PDF.

Generalizability and validity of the evaluation

We wish to clarify that our primary goal is to serve as a cornerstone research for evaluating cross-modal hallucinations, i.e., assessing how one modality might affect the perception of another. While our dataset, built upon VALOR and AudioCaps, may not encompass all real-world scenarios, it does cover a diverse range of in-the-wild scenarios related to human actions, mechanics, nature, and animal-related events, as illustrated in Figure S8 in the Appendix of the initial submission. As the original purposes and generalities of each existing dataset have already been acknowledged by their own, we believe that their union coverage in our work is still valid to evaluate such hallucinations in audio-visual LLMs. Nevertheless, we concur that our dataset may not cover all scenarios and including more scenarios and datasets would make this benchmark more comprehensive. This point has already been discussed in "Section 5. Conclusion and Discussion" of our initial submission, and we leave more extensions for future work.

Content of speech may introduce hallucination

We agree that speech (linguistic content) may induce hallucinations and recognize that addressing this could be an entirely new impactful direction for research, making the dataset more comprehensive. One approach could involve integrating speech content into our dataset construction pipeline through Automatic Speech Recognition (ASR). This addition would help explore how speech influences visual perception hallucinations, but it becomes a completely different research. We have newly added Section 5 Limitation and future work in the revision PDF, and included this discussion in Line 485-504. We’d like to highlight that, as acknowledged by all the reviewers, our benchmark focuses on the scope that has barely been explored and is still valuable in the community. As mentioned in Sec. 3.2. in our initial submission, our benchmark focuses on environmental- or event-level videos and audios. While few studies dealing with both speech and environment sounds have recently begun, the vast majority of audio-video language models still rely on audio encoders only dealing with environmental audio, not speech. We’d like to ask the reviewer to recognize this contribution.

Evaluation on more recent audio-visual models

As requested by reviewer huna, we have conducted additional evaluations on our benchmark with 5 more recent models; audio-visual LLMs, including Video-LLaMA2, Video-SALMONN, and Gemini, and unimodal LLMs, including LLaVA-Onevision, and Qwen2-audio. The results are summarized in the provided table and these new results and discussions are newly updated in Appendix Section C. Additional quantitative results of the revision PDF.

We observe that overall performance on the benchmark has improved with these newer models compared to those used in our main paper. Interestingly, we find that overall model performance improves when using unimodal signals for evaluation (comparing results from Audio-driven Video Hallucination vs. Video Hallucination; Video-driven Audio Hallucination vs. Audio Hallucination). Specifically, Video-LLaMA2 demonstrates notable improvements with unimodal signals compared to using both audio-visual signals, supporting discussions in Section 4.2 of the initial submission on how cross-modal signals can confuse models and evoke hallucinations. These findings highlight recent enhancements in models to address cross-modal hallucination and also demonstrate the effectiveness of our benchmark in evaluating these advancements.

Furthermore, we plan to maintain a benchmark leaderboard webpage to facilitate ongoing evaluation of audio-visual LLMs. We have newly updated this captured photo of the leaderboard in Appendix Section H Maintenance of the revision PDF.

We thank reviewer huna for their valuable feedback. However, we respectfully request that reviewer huna reconsider our work, which focuses distinctly on evaluating hallucinations in concurrent generic audio-visual signals. This research serves as a cornerstone for studies on audio-visual cross-modal hallucinations, with the potential to extend to speech in the future. Our main focus aligns with recent advancements in audio-visual LLMs that process generic audio inputs rather than speech (e.g., X-InstructBLIP, ImageBind-LLM, Video-LLaMA, Video-LLaMA2, ChatBridge, PandaGPT, OneLLM, LTU). Speech-integrating MLLMs, such as Video-Salmonn, have only recently been proposed. While we agree that speech is an important auditory signal, generic audio is also very crucial for evaluating the audio-visual understanding of MLLMs. Involving datasets like HowTo100M or VideoChat to address speech hallucination would indeed be valuable but represent a completely different research direction. We respectfully request reviewer huna reassess our work according to the scope of generic audio rather than introducing an out-of-scope context. Thank you.

We thank reviewer 7vLX for recognizing our work as providing the first comprehensive benchmark for audio-visual LLMs, well-designed figures, and clear organization, along with constructive feedback. Here we address the concerns raised by reviewer 7vLX, and encourage the reviewer 7vLX to check the newly updated revision PDF that includes the requested content. The red colored texts indicate the newly added part in the revision PDF. We respectfully request the reviewer 7vLX to reassess our work and consider increasing the rating.

Increasing the number of evaluation models

As requested by reviewer 7vLX, we have conducted additional evaluations on our benchmark with 5 more recent models; audio-visual LLMs, including Video-LLaMA2, Video-SALMONN, and Gemini, and unimodal LLMs, including LLaVA-Onevision, and Qwen2-audio. The results are summarized in the provided table and these new results and discussions are newly updated in Appendix Section C. Additional quantitative results of the revision PDF.

We observe that overall performance on the benchmark has improved with these newer models compared to those used in our main paper. Interestingly, we find that overall model performance improves when using unimodal signals for evaluation (comparing results from Audio-driven Video Hallucination vs. Video Hallucination; Video-driven Audio Hallucination vs. Audio Hallucination). Specifically, Video-LLaMA2 demonstrates notable improvements with unimodal signals compared to using both audio-visual signals, supporting discussions in Section 4.2 of the initial submission on how cross-modal signals can confuse models and evoke hallucinations. These findings highlight recent enhancements in models to address cross-modal hallucination and also demonstrate the effectiveness of our benchmark in evaluating these advancements.

Furthermore, we plan to maintain a benchmark leaderboard webpage to facilitate ongoing evaluation of audio-visual LLMs. We have newly updated this captured photo of the leaderboard in Appendix Section H Maintenance of the revision PDF.

Limitation in evaluating a broader range of real-world scenarios

Although our dataset does not encompass every real-world scenario (we have acknowledged limitations and have already described it in "Section 5. Conclusion and Discussion"), we believe the diversity is reasonable for our dataset as an important cornerstone research for evaluating cross-modal hallucinations in audio-visual LLMs. Our dataset includes a broad range of in-the-wild situations involving human actions, mechanics, nature, and animal-related events, as shown in Figure S8 in the Appendix. This is a virtue of our benchmark constructed by integrating multiple existing datasets; thus, our benchmark is much more diverse than the test subsets of each individual dataset. Moreover, as mentioned in Section D.7 of the Appendix, we are expanding the dataset after the main submission to incorporate even more diverse scenarios.

Furthermore, please refer to the response to reviewer MmyJ, where we have added discussions on diverse scenarios, such as ensuring visuals or audio feature objects that are silent or absent, handling ambient sounds like wind, managing multiple objects of the same type with different statuses, and addressing background music. These points highlight that our dataset already captures a wide variety of meaningful and challenging aspects.

To avoid confusion and better address concerns, it would be helpful if you could provide specific suggestions on the aspects that should be included to diversify the scenarios.

We thank reviewer FJwk for recognizing our work as proposing a valuable benchmark, reasonable semi-automatic pipeline, and well-written with clear motivation, along with constructive feedback. We address the concerns raised by reviewer FJwk, and encourage the reviewer FJwk to check the newly updated revision PDF that includes the requested content.

More evaluation on more recent audio-visual models

As requested by reviewer FJwk, we have conducted additional evaluations on our benchmark with 5 more recent models; audio-visual LLMs, including Video-LLaMA2, Video-SALMONN, and Gemini, and unimodal LLMs, including LLaVA-Onevision, and Qwen2-audio. The results are summarized in the provided table and these new results and discussions are newly updated in Appendix Section C. Additional quantitative results of the revision PDF.

We observe that overall performance on the benchmark has improved with these newer models compared to those used in our main paper. Interestingly, we find that overall model performance improves when using unimodal signals for evaluation (comparing results from Audio-driven Video Hallucination vs. Video Hallucination; Video-driven Audio Hallucination vs. Audio Hallucination). Specifically, Video-LLaMA2 demonstrates notable improvements with unimodal signals compared to using both audio-visual signals, supporting discussions in Section 4.2 of the initial submission on how cross-modal signals can confuse models and evoke hallucinations. These findings highlight recent enhancements in models to address cross-modal hallucination and also demonstrate the effectiveness of our benchmark in evaluating these advancements.

Furthermore, we plan to maintain a benchmark leaderboard webpage to facilitate ongoing evaluation of audio-visual LLMs. We have newly updated this captured photo of the leaderboard in Appendix Section H Maintenance of the revision PDF.

We thank reviewer MmyJ for recognizing our work as well-written, well-motivated, and for providing a comprehensive benchmark, valuable takeaways, and methods for improving the trustworthiness of MLLMs, along with constructive feedback. Here, we address the concerns raised by reviewer MmyJ.

Ensuring the quality of the dataset

We'd like to emphasize that our dataset is human-verified in all dataset construction stages, which ensures its quality. Our proposed data generation method effectively transforms the exhaustive human labeling process into a focused verification effort. Thus, our resulting dataset is at the level of humans. As mentioned in Section D.4 in the Appendix of our initial submission, we assigned different tasks to each annotator and used majority votes to finalize each QnA label. The first annotator checked the Q&As and audio-visual captions generated with automatic annotations. The second annotator was presented with 10% intentionally inaccurate QnAs (verified by the authors), and successfully revised 9.8% of them. The last annotator annotated everything from scratch. During these phases, 94.4% of the QnA pairs were agreed upon by all three annotators, while 6.6% were agreed upon by two. This process helped to ensure the quality of our dataset. Additionally, we have newly included new error analyses in Section D.5 and Figure S6 of the Appendix in the revision PDF. These sections illustrate how errors are corrected through human verification, further ensuring the dataset's quality.

Ensuring the visuals or audio contains objects that are either silent or not present

We recognize that not all videos contain silent objects or out-of-view sound sources (which was also mentioned by Reviewer MmyJ) and already discussed this in Section 3.2 (Lines 232-235) of our initial submission. Consequently, the videos that contain these cases are used for generating QnAs related to hallucinations caused by in-view silent objects or out-of-view sound sources, while others are used to provide QnAs for specific tasks. In addition, our Stage 1 audio-visual information disentanglement, followed by human verification, ensures the accuracy of extracting silent visuals and non-present sound sources. As mentioned in lines 239-255 of our initial submission, ChatGPT disentangles in-view and out-of-view information using provided audio captions and visual tags, with human-annotated few-shot examples guiding the process. Furthermore, human verification is conducted to correct any inaccuracies in disentangled information, as detailed in lines 266-271 of the initial submission. If neither in-view silent objects nor out-of-view sound sources exist, they are annotated as "none."

How can ambient sounds (e.g., wind) be handled?

While not specifically handled, questions that do not make sense (e.g., “Is the wind visible?”) are filtered out by human annotators who checked such queries as “ambiguous” as shown in Figure S5 in the Appendix of the initial submission. However, these ambient sounds are annotated in Stage 1 and can be used to extend our dataset for a single-modal hallucination benchmark, such as Audio-only hallucination.

Handling multiple objects with the same type but different statuses?

Involving “events” helped to handle multiple objects of the same type with different statuses. For instance, if an in-view person is silent while there is speech happening out-of-view, our Stage 1 pipeline will annotate “person” for “in-view silent object,” and “talking person” for “out-of-view sound source.” Therefore, when generating QnAs in Stage 2, it might be phrased as “Is a talking person visible in the video? No.” Since the person exists in the video but is not talking, this question serves as a hallucination QnA. Furthermore, all annotations in both Stage 1 and Stage 2 are verified by humans, who correct any nonsensical QnAs that were automatically generated by our proposed pipeline.

Handling inconsistent audio and video signals (e.g., background music)

Although inconsistencies between video and audio from the same source may not affect the construction of cross-modal hallucination pairs, they become critical when managing matching tasks, such as when background music is unrelated to the visual scene. We handle these inconsistencies through human verification. These sources are automatically assumed to be “matched” since they originate from the same video; however, if human annotators consider them unaligned, they adjust the answer accordingly. For instance, in videos with background music, annotators label these as inconsistent and classify them as "not matching," or they label them as "ambiguous" if they are difficult to determine. Finally, we take the majority vote from three annotators to decide the labels for these matchings, thus managing or discarding inconsistent audio-video signals effectively.

We thank reviewer sQPA for recognizing our work as novel, well-written, and clearly motivated, along with providing constructive feedback. We address the concerns raised by sQPA below and encourage the reviewer sQPA to check the newly updated revision PDF that includes the requested content. The red-colored texts indicate the newly added part in the revision PDF.

Error analysis

We appreciate reviewer sQPA for suggesting us to conduct an error analysis to improve our paper. First of all, we’d like to highlight that our final data is human-verified data, and initial errors should be corrected by humans. Our proposed data generation method is actually an efficient and scalable human annotation tool that converts from an exhaustive human labeling process to a focused verification one. Thus, our data is at the level of humans. Nonetheless, according to the request, we have analyzed and revealed how much human verification plays a crucial role in identifying and correcting errors that occur in automatically generated annotations. We have newly added this error analysis in Section D.5 and Figure S6 in the Appendix of the revision PDF and provided the details below.

(1) (audio-visual disentanglement) Error in the visual tagging

• Errors in visual tagging are handled by Stage 1 human annotators. As we use visual tagging extracted from an off-the-shelf model to disentangle audio-visual information, errors are sometimes included. However, as shown in Figure S6-(a) in the Appendix of the revision PDF, human annotators review all disentangled information and correct any incorrect disentangled visual information. For example, the automatically disentangled in-view silent object "plane," which does not exist in the video but is captured in the visual tagging, is successfully removed by the annotators.

(2) (audio-visual disentanglement) Distinguishing multiple instances of the same type of object

• Our work primarily serves as a corner research that first evaluates and reveals cross-modal hallucinations, i.e., assessing how one modality might affect the perception of another. We mainly focus on scenarios where out-of-view sound sources affect visual perception in audio-visual LLMs, or in-view silent objects affect audio perception. Thus, our evaluation does not include the fine-grained analysis that the reviewer mentioned, such as distinguishing which of two visible people made a sound. However, we agree that incorporating this finer evaluation into future work could make our benchmark more comprehensive. We have newly added Section 5 Limitation and future work in the revision PDF, and included this discussion in Line 475-479.

(3) (audio-visual caption generation) Capturing correspondence between unaligned visual and audio captions

• Human verification is also important in handling inaccuracies in the initial audio-visual captions due to unaligned visual and audio captions. During this rebuttal period, our analysis revealed that 33% of captions were corrected by human annotators. Although our automatic pipeline effectively predicted 67% of the captions accurately, human verification proved essential for ensuring accuracy. An example highlighting the necessity of this verification can be seen in the newly added Figure S6-(b) in the Appendix of the revision PDF. Here, the given audio caption describes, "A woman speaking as water softly splashes and trickles while wind lightly blows into a microphone," while the video caption incorrectly states, "A man sitting on a large rock in the middle of a lake," as the video shows a woman sitting on the rock. The initial audio-visual caption generated by ChatGPT, therefore, inaccurately merges these descriptions, resulting in: "A woman speaks softly as water splashes and trickles with a man sitting on a large rock in the middle of a lake." However, human verification identifies this error, and corrects it from “with a man” to “and she is,” ensuring the refined audio-visual caption accurately describes the video: “A woman speaks softly as water splashes and trickles, and she is sitting on a large rock in the middle of a lake.”

Thanks for the suggestion, which improves the completeness of our work.

Dear Reviewer sQPA,

We sincerely appreciate reviewer sQPA’s time and effort in reviewing our submissions and providing valuable feedback. We believe all the comments and experiments suggested by the reviewer sQPA have been addressed, including the error analyses and additional evaluation on another benchmarks. We welcome any further comments or feedback that may help to enhance our work. We'd be happy to further discuss. Thank you.